Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [.........................................XXX.XXXXXXXXXXXXXXX] duration: 1m0.062333332s FAIL (2025-07-28 00:23:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== : Uploading piece... Jul 28 00:22:41.567962568 (26h30m57.348895954s) [noise=true , address=104.36.82.3:4770 , piece_id=REZANEVMXYIR5HQCH4ZAQHDCLRX577PDQHSAT4XGCV3LGBYGCX4A, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.569497158 (26h30m57.350430544s) [noise=true , address=104.36.82.52:4119 , piece_id=YBWGOCDLVRWX5ATL5SA5COXEIRZ5Z4NX7GDZVRSRK2DJCBPVNZJA, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.570220621 (26h30m57.351154003s) [noise=true , address=104.36.82.53:4134 , piece_id=JS2FO7B5OWRWNVZVJXEEXGCERNJOB5QOEUGUXVU7EZ4VYMHXV3VQ, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.571346077 (26h30m57.352279464s) [noise=true , address=104.36.82.46:4502 , piece_id=36QF4EQS62L34VRP6X4TUKL7PCDZZF3JA3B6LMPNASHRYHCNTFHQ, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.571810505 (26h30m57.352743885s) [noise=true , address=104.36.82.55:4201 , piece_id=FJLWAFMFMTLOAJ5AYJZD5WRZ5XC3WO6PXL6ZQ64TGWTDJK2X6FPA, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.572394285 (26h30m57.353327667s) [noise=true , address=104.36.82.20:4459 , piece_id=W3BOCNUVVXZP2YXNG354PZSCK4ZOUCFFJAGZPUJSKDDQZGZALQ2Q, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.572830490 (26h30m57.353763873s) [noise=true , address=104.36.82.47:4543 , piece_id=QWT7IB6ALL3EOQNP4XMFJVORLNWNR7SRFKKEPNDGHIR2UEU6ESKQ, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.573268094 (26h30m57.354201482s) [noise=true , address=104.36.82.14:4228 , piece_id=HM55DPNXQ7K6HCP7PGMATEDGZCNOQW26R6K5VMX426YF3OEY7ISQ, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.573683414 (26h30m57.354616799s) [noise=true , address=104.36.82.12:4180 , piece_id=OIZPDYBZ6Z46CMNNLSCWBA4DBZIECTGXHM7ICXYXSGPN3V365SQA, seg_pos={0 0}, upload=25309]: Uploading piece... Jul 28 00:22:41.773343829 (26h30m57.554277212s) [upload=25309]: Next returned nil segment Jul 28 00:22:41.773367760 (26h30m57.554301145s) [upload=25309]: Waiting for error group managing segments... Jul 28 00:22:46.743972499 (26h31m2.524905897s) [noise=true , address=77.90.31.16:4744 , piece_id=IZRQI67AXTSVX6FUORLWEWGSHAGCTQL6XS6Y46F7XYT7YGREWL6Q, seg_pos={0 0}, upload=25309]: Done uploading piece. err: <nil> Jul 28 00:22:47.664618143 (26h31m3.445551518s) [noise=true , address=79.127.201.215:4360 , piece_id=SNER7ANSCR5WPOSBM3FC5J7VOFR3BI3JFJONQLF3X6TMFZXW2FZA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: <nil> Jul 28 00:23:41.496665723 (26h31m57.277599103s) [noise=true , address=104.36.82.47:4547 , piece_id=ST32IUP7KZYZ5JXB6XFHRJM6QWLIZJIKE7Q24IRE5AJOFBQTLTPA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12brQg9WQMKJgp1buz1wMS8W11XEwAx7RHG5h51EmLvwY5D5Hky, address:104.36.82.47:4547): protocol: rpc: dial tcp 104.36.82.47:4547: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.496761937 (26h31m57.277695313s) [noise=true , address=104.36.82.15:4288 , piece_id=L5U6GCRT3CCHLYWWHY3NJOYHGJC2ESCRHWYLUTP56IWXRCYXE5AQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7, address:104.36.82.15:4288): protocol: rpc: dial tcp 104.36.82.15:4288: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.496858906 (26h31m57.277792282s) [noise=true , address=104.36.82.12:4180 , piece_id=OIZPDYBZ6Z46CMNNLSCWBA4DBZIECTGXHM7ICXYXSGPN3V365SQA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: rpc: dial tcp 104.36.82.12:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.496931683 (26h31m57.277865059s) [noise=true , address=104.36.82.46:4502 , piece_id=36QF4EQS62L34VRP6X4TUKL7PCDZZF3JA3B6LMPNASHRYHCNTFHQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.496993364 (26h31m57.277926739s) [noise=true , address=104.36.82.13:4202 , piece_id=KBHUZJLB5KHRRF6RKHLO4S6MIJ3YRFZV7FINM2VITYXVQBNQATJA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:18wf5KnaR6WaR7MJivJkMJ7FejbuR6KnyvtRQgazNKfDb8fbsH, address:104.36.82.13:4202): protocol: rpc: dial tcp 104.36.82.13:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497028360 (26h31m57.277961786s) [noise=true , address=104.36.82.45:4929 , piece_id=VCWYU3LV2FXQN74OGTVXP2WR34FTFTF5ZY22P5FPZSUSWHQGVPKQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm, address:104.36.82.45:4929): protocol: rpc: dial tcp 104.36.82.45:4929: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497075291 (26h31m57.278008666s) [noise=true , address=104.36.82.44:4904 , piece_id=TG4AI2RT63XP6R65YVG6IDCO63TD2CUZXNWNM2EVOVC2FVSXB72A, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC, address:104.36.82.44:4904): protocol: rpc: dial tcp 104.36.82.44:4904: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497146365 (26h31m57.278079753s) [noise=true , address=104.36.82.3:4770 , piece_id=REZANEVMXYIR5HQCH4ZAQHDCLRX577PDQHSAT4XGCV3LGBYGCX4A, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:104.36.82.3:4770): protocol: rpc: dial tcp 104.36.82.3:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497219658 (26h31m57.278153034s) [noise=true , address=104.36.82.21:4494 , piece_id=VCC54HWJ2OZES47SN7KCS7MRAWUY4QZPU2MSAJUMHJ4YL522ZH6Q, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1sRaMuQG18B5SfPpzxp5VWjrwe38SS38g4askmbURxU9tepJHi, address:104.36.82.21:4494): protocol: rpc: dial tcp 104.36.82.21:4494: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497299255 (26h31m57.278232643s) [noise=true , address=104.36.82.53:4134 , piece_id=JS2FO7B5OWRWNVZVJXEEXGCERNJOB5QOEUGUXVU7EZ4VYMHXV3VQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: rpc: dial tcp 104.36.82.53:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497369485 (26h31m57.278302858s) [noise=true , address=104.36.82.20:4459 , piece_id=W3BOCNUVVXZP2YXNG354PZSCK4ZOUCFFJAGZPUJSKDDQZGZALQ2Q, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:1hpq5G467oUtEaE6Cj842pHCCZ1V7EraP5EDdGdWMxz4M3Q3pA, address:104.36.82.20:4459): protocol: rpc: dial tcp 104.36.82.20:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497397688 (26h31m57.278331067s) [noise=true , address=104.36.82.55:4201 , piece_id=FJLWAFMFMTLOAJ5AYJZD5WRZ5XC3WO6PXL6ZQ64TGWTDJK2X6FPA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497428141 (26h31m57.278361528s) [noise=true , address=104.36.82.47:4543 , piece_id=QWT7IB6ALL3EOQNP4XMFJVORLNWNR7SRFKKEPNDGHIR2UEU6ESKQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12KxnBTVxhbwA79Gd8V7Mh6bWybtqttqgo5agHoHewXHAey5y7D, address:104.36.82.47:4543): protocol: rpc: dial tcp 104.36.82.47:4543: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497441107 (26h31m57.278374471s) [noise=true , address=104.36.82.52:4119 , piece_id=YBWGOCDLVRWX5ATL5SA5COXEIRZ5Z4NX7GDZVRSRK2DJCBPVNZJA, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12mTLjL6Y6VVjqztwVXYif4v7tyftjLafztLUibajksMHCFraFG, address:104.36.82.52:4119): protocol: rpc: dial tcp 104.36.82.52:4119: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497454182 (26h31m57.278387545s) [noise=true , address=104.36.82.15:4274 , piece_id=NP6WPSZVJRKRP2BWZOBD5KXK6QZIBS4A2GBRZIQ3U6AGORQE2UBQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12nZ58ntQujpYk3UVGt4rD6xssGW3RxmvgM84a4Ka8BCRowuaim, address:104.36.82.15:4274): protocol: rpc: dial tcp 104.36.82.15:4274: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497462919 (26h31m57.278396281s) [noise=true , address=104.36.82.14:4228 , piece_id=HM55DPNXQ7K6HCP7PGMATEDGZCNOQW26R6K5VMX426YF3OEY7ISQ, seg_pos={0 0}, upload=25309]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: write tcp 159.223.143.39:37846->104.36.82.14:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:23:41.497507499 (26h31m57.278440884s) [seg_pos={0 0}, upload=25309]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:23:41.556138464 (26h31m57.337071835s) [upload=25309]: Done uploading segments... Jul 28 00:23:41.556163895 (26h31m57.337097264s) [upload=25309]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.062370803s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X...XX.......X..] duration: 529.188691ms FAIL (2025-07-28 00:23:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 529.216001ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X...XX.......X.] duration: 470.021029ms FAIL (2025-07-28 00:23:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 470.049655ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X........X........] duration: 444.152436ms FAIL (2025-07-28 00:22:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.196671ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X...XX.......X] duration: 30.026525096s FAIL (2025-07-28 00:22:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.026546783s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 348.546741ms FAIL (2025-07-28 00:22:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 348.58552ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 17 out of 60 is failed (threshold=3) [..........................................XXX.XXXXXXXXXXXXXX] duration: 1m0.040319572s FAIL (2025-07-28 00:22:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 25290]: Uploading piece... Jul 28 00:21:41.483328084 (26h29m57.264261467s) [noise=true , address=104.36.82.46:4515 , piece_id=H33YBP3JDSIKRD24YHTS6U6VN6NE5ZXLD5TJBWP3QJTMGDE2UF3A, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.485565755 (26h29m57.266499142s) [noise=true , address=104.36.82.55:4198 , piece_id=33EFCGEMN3PZ2U3PU6QNI6DLM7GAO3VVCMU4T6RZSYXRP4E3XE3Q, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.487630081 (26h29m57.268563457s) [noise=true , address=104.36.82.53:4148 , piece_id=4WZJKIOM3CQARHAC5J54FEM4UXQ6OI72PZI3PWCWWHBDIBHUR3CQ, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.488025843 (26h29m57.268959225s) [noise=true , address=104.36.82.10:4089 , piece_id=7DZ3ZR2WR3NQBGRDUPIPRIPR4ZOCIMLUK25GP3V2LGNRWOXKIRLQ, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.489924722 (26h29m57.27085811s) [noise=true , address=104.36.82.21:4478 , piece_id=DC5YA6BSCF772G4V55TAY2T3EPVGHLIT4OZOHSSRROYQDKNZVB2Q, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.490541848 (26h29m57.271475234s) [noise=true , address=104.36.82.4:4836 , piece_id=SJOG7BPO4O3GWUHBCPKGUZER7J2J6KL67QVFGEMHGPSG5Q66RFDA, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.491926865 (26h29m57.272860253s) [noise=true , address=79.127.201.215:4353 , piece_id=ZWEG2SOV23FK5KS4NNIV3WEQVVTYFMAOAKCJIRE5OFEWGWVBQJQQ, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.492590382 (26h29m57.273523764s) [noise=true , address=79.127.205.234:4619 , piece_id=ONQ6DOMMYONHC7K33WL2HAM6PEJKIGPMJKR7WRPY22XOFLHXR3FA, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.494297212 (26h29m57.275230592s) [noise=true , address=104.36.82.13:4191 , piece_id=T5V4GNFULI4GHYK5BS7KSENNICT2DJI5DNFPVA36IETYSF45Z3DA, seg_pos={0 0}, upload=25290]: Uploading piece... Jul 28 00:21:41.686457009 (26h29m57.467390387s) [upload=25290]: Next returned nil segment Jul 28 00:21:41.686481493 (26h29m57.467414872s) [upload=25290]: Waiting for error group managing segments... Jul 28 00:21:42.210589749 (26h29m57.991523119s) [noise=true , address=79.127.205.234:4619 , piece_id=ONQ6DOMMYONHC7K33WL2HAM6PEJKIGPMJKR7WRPY22XOFLHXR3FA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: <nil> Jul 28 00:21:43.685485742 (26h29m59.466419107s) [noise=true , address=79.127.201.215:4353 , piece_id=ZWEG2SOV23FK5KS4NNIV3WEQVVTYFMAOAKCJIRE5OFEWGWVBQJQQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: <nil> Jul 28 00:22:41.391099061 (26h30m57.172032437s) [noise=true , address=104.36.82.55:4198 , piece_id=33EFCGEMN3PZ2U3PU6QNI6DLM7GAO3VVCMU4T6RZSYXRP4E3XE3Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12YxL4YR6mNE536w1SnBrMUZHiaRTuU68Zjg9LBvRURVx5s5HWL, address:104.36.82.55:4198): protocol: rpc: dial tcp 104.36.82.55:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391487218 (26h30m57.17242062s) [noise=true , address=104.36.82.46:4504 , piece_id=RSC54PFOYEFGK6PSHHI7K746VUE3FS5TO2CC7IAXQQJ7XARQX4HA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: rpc: dial tcp 104.36.82.46:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391513090 (26h30m57.172446466s) [noise=true , address=104.36.82.11:4132 , piece_id=RGBNEXDA4FEDPNFQZIKJU6NOU27W7XNQ2HNKVR254VGG6EKW2CRA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391596103 (26h30m57.172529477s) [noise=true , address=104.36.82.53:4148 , piece_id=4WZJKIOM3CQARHAC5J54FEM4UXQ6OI72PZI3PWCWWHBDIBHUR3CQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391668207 (26h30m57.172601583s) [noise=true , address=104.36.82.21:4478 , piece_id=DC5YA6BSCF772G4V55TAY2T3EPVGHLIT4OZOHSSRROYQDKNZVB2Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:17kFjjJyvHApPUxUHJ6HeMEBCoo6xTbr8QizXjBEHwWj3mZUha, address:104.36.82.21:4478): protocol: rpc: dial tcp 104.36.82.21:4478: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391676613 (26h30m57.172609991s) [noise=true , address=104.36.82.46:4515 , piece_id=H33YBP3JDSIKRD24YHTS6U6VN6NE5ZXLD5TJBWP3QJTMGDE2UF3A, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: rpc: dial tcp 104.36.82.46:4515: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391737703 (26h30m57.172671077s) [noise=true , address=104.36.82.12:4160 , piece_id=YV425YTIHBFDSGBM4TCDLIFPUXE2MT3EWQM3E4TGWPD4APSGIE6Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12dggZZRv5coYBNJEBvHNqWtMSqYkZoXYmitc9nBKJZftnsX17D, address:104.36.82.12:4160): protocol: rpc: dial tcp 104.36.82.12:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391768098 (26h30m57.172701475s) [noise=true , address=104.36.82.42:4831 , piece_id=6X2DZFR2FJCUAS2EEMKJML3PLPI3YT43JQP6UOIWXFSILG4FWFTA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: rpc: dial tcp 104.36.82.42:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391838699 (26h30m57.172772077s) [noise=true , address=104.36.82.47:4535 , piece_id=Y3PQAYUI2W4CRM3YV2OV4LCRTJYEO4EKGZTZUJ36IRE524UNC22Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391856126 (26h30m57.172789507s) [noise=true , address=104.36.82.2:4742 , piece_id=S3XSIA4P3YH2GQLGXBVJOAYWSYLVYM44CIAO6WV347GMKREXU67Q, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1WsGo4te3PENFo9YRioYFrhkeRQvztnLVZJV4i38TV3TjKqBFT, address:104.36.82.2:4742): protocol: rpc: dial tcp 104.36.82.2:4742: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.391957135 (26h30m57.17289051s) [noise=true , address=104.36.82.13:4191 , piece_id=T5V4GNFULI4GHYK5BS7KSENNICT2DJI5DNFPVA36IETYSF45Z3DA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1sVWXyKPnLTobAhk6GXZTTP67FbyY3XJ1e2svTiUEa3rrghd2q, address:104.36.82.13:4191): protocol: rpc: dial tcp 104.36.82.13:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392034885 (26h30m57.172968399s) [noise=true , address=104.36.82.54:4173 , piece_id=PB7GRW56RRIMQGE3K75I2QJ7DGWVK34ZQKSDPQLAPGMLDN23SNGA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: rpc: dial tcp 104.36.82.54:4173: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392044423 (26h30m57.172977792s) [noise=true , address=104.36.82.14:4234 , piece_id=ZX6SN77U4F5E5UN6Q7CRCU7PJVWS47H7HZIBDOII7OPSTC4U6AJQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12DtCsEnAhXZnGRXjnhNfqNz4RvPnvTMDz1ur7DyZiQtQ7nbwDS, address:104.36.82.14:4234): protocol: rpc: dial tcp 104.36.82.14:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392056095 (26h30m57.172989472s) [noise=true , address=104.36.82.40:4950 , piece_id=YA2GKHJ5RHSXJ7KGAGRLKKYW5OXT44IBQJ6FOKBLKGTCPGVS2JHQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1Y6fZC1qCRQYMEvW3475SvnaWYCidxcDhch2YA1ZXW5U8fXDZk, address:104.36.82.40:4950): protocol: rpc: dial tcp 104.36.82.40:4950: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392099925 (26h30m57.173033301s) [noise=true , address=104.36.82.10:4089 , piece_id=7DZ3ZR2WR3NQBGRDUPIPRIPR4ZOCIMLUK25GP3V2LGNRWOXKIRLQ, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: rpc: dial tcp 104.36.82.10:4089: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392130596 (26h30m57.173063976s) [noise=true , address=104.36.82.4:4836 , piece_id=SJOG7BPO4O3GWUHBCPKGUZER7J2J6KL67QVFGEMHGPSG5Q66RFDA, seg_pos={0 0}, upload=25290]: Done uploading piece. err: ecclient: upload failed (node:1fjbyyAnz2pYU8drcBLyd1VzHgsNSWAxdWXovYJWzTcSdANzNe, address:104.36.82.4:4836): protocol: write tcp 159.223.143.39:44864->104.36.82.4:4836: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:22:41.392158883 (26h30m57.173092268s) [seg_pos={0 0}, upload=25290]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:22:41.427412168 (26h30m57.20834553s) [upload=25290]: Done uploading segments... Jul 28 00:22:41.427443367 (26h30m57.20837673s) [upload=25290]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.040342103s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X........X.......] duration: 450.728661ms FAIL (2025-07-28 00:22:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.755986ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X........X......] duration: 572.916132ms FAIL (2025-07-28 00:22:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 572.951057ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 229.722868ms FAIL (2025-07-28 00:21:55 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 229.763432ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 16 out of 60 is failed (threshold=3) [...........................................XXX.XXXXXXXXXXXXX] duration: 1m0.025682029s FAIL (2025-07-28 00:21:41 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.371192168 (26h28m57.152125549s) [noise=true , address=104.36.82.21:4503 , piece_id=BCZTHVIZMU6PPGUU75TYN2XC6VB2JQYBIMMFZLKBU6A73OAUABYA, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.371525656 (26h28m57.152459038s) [noise=true , address=104.36.82.53:4154 , piece_id=MOLL2F7FKVDS2FZDA2CC7KVHOFSIHG5EBPEM6VU5ZHOSWQTYZQRA, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.371849912 (26h28m57.152783291s) [noise=true , address=104.36.82.55:4196 , piece_id=YBSM5ABU4IUSUZ44I2VHWTPVZY6OGBYEZA6FZT2MX6KQHHGARWPQ, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.372252300 (26h28m57.153185691s) [noise=true , address=104.36.82.15:4279 , piece_id=EL7MFXVWOG2SRXX4RA3EXKYJB52TKLUWDHXYEMIJRIC5T525DCMQ, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.372669809 (26h28m57.153603191s) [noise=true , address=104.36.82.2:4755 , piece_id=ZIWQFL2CLETGAQXC4UOPOE7T3Y632KF5RZB5ZUBRX3M6XVLOJF5Q, seg_pos={0 0}, upload=25273]: Uploading piece... Jul 28 00:20:41.854332036 (26h28m57.635265398s) [upload=25273]: Next returned nil segment Jul 28 00:20:41.854358717 (26h28m57.635292095s) [upload=25273]: Waiting for error group managing segments... Jul 28 00:20:44.216778103 (26h28m59.997711485s) [noise=true , address=77.90.31.16:4748 , piece_id=X336XIAHIYHCFUY2HHBUZ356W4TAV5XEJFZ3OMNXFDA42QH2A6YA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: <nil> Jul 28 00:21:41.293343579 (26h29m57.074276965s) [noise=true , address=104.36.82.15:4279 , piece_id=EL7MFXVWOG2SRXX4RA3EXKYJB52TKLUWDHXYEMIJRIC5T525DCMQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293421082 (26h29m57.074354452s) [noise=true , address=104.36.82.54:4187 , piece_id=7OJZPBJDU2JSHLUA674SBJP3VLE2EHZ3ZZY3XKEY3QFVJ7J6U55Q, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, address:104.36.82.54:4187): protocol: write tcp 159.223.143.39:47344->104.36.82.54:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293614435 (26h29m57.07454781s) [noise=true , address=104.36.82.11:4131 , piece_id=TJZQJHUZD4KDRUJ7BGFHJLA7WQAURQNUUNJQAEWEEVAJHFV6PWAA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:129jahwAmyPHqyYaXz7XTCK8FakBrYZw94pBKGhPzzX23yjM3RQ, address:104.36.82.11:4131): protocol: write tcp 159.223.143.39:47068->104.36.82.11:4131: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293715157 (26h29m57.074648533s) [noise=true , address=104.36.82.21:4503 , piece_id=BCZTHVIZMU6PPGUU75TYN2XC6VB2JQYBIMMFZLKBU6A73OAUABYA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:17F3Gn5LYgekp8WgagEdFXtRu8jCc27qA9UuW7pWsJ9FUFKFSi, address:104.36.82.21:4503): protocol: write tcp 159.223.143.39:41764->104.36.82.21:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293773285 (26h29m57.074706674s) [noise=true , address=104.36.82.53:4154 , piece_id=MOLL2F7FKVDS2FZDA2CC7KVHOFSIHG5EBPEM6VU5ZHOSWQTYZQRA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: write tcp 159.223.143.39:56510->104.36.82.53:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293838795 (26h29m57.07477217s) [noise=true , address=104.36.82.9:4022 , piece_id=MVCAEDRHX5NDCOOQ3KSV3YUTA6X7P3TGRQBGQF6JXRBIIIDEQMJA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1XGBWwhqJ324UTpv3b1bNsGVUjJSLptvqV6dHrYJsto8KbDvYU, address:104.36.82.9:4022): protocol: write tcp 159.223.143.39:50654->104.36.82.9:4022: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293938568 (26h29m57.074871943s) [noise=true , address=104.36.82.55:4196 , piece_id=YBSM5ABU4IUSUZ44I2VHWTPVZY6OGBYEZA6FZT2MX6KQHHGARWPQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: write tcp 159.223.143.39:34154->104.36.82.55:4196: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.293997001 (26h29m57.074930379s) [noise=true , address=104.36.82.2:4755 , piece_id=ZIWQFL2CLETGAQXC4UOPOE7T3Y632KF5RZB5ZUBRX3M6XVLOJF5Q, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1NesDEUEmNTHsmSYb1V6esyDAryke6A7NmRnPE6mwPknTfUFF3, address:104.36.82.2:4755): protocol: write tcp 159.223.143.39:43770->104.36.82.2:4755: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294068450 (26h29m57.075001825s) [noise=true , address=104.36.82.55:4227 , piece_id=PKEMZXJ25F4HSX7B6X4JTYLEIQRWIALETKKLSBM3GUKFVHR47AZA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:19qGGwWT9xUYsJyRnFHyHKPFr6g4AsckxRNVTg2shfVhoQtcCz, address:104.36.82.55:4227): protocol: write tcp 159.223.143.39:36152->104.36.82.55:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294296983 (26h29m57.075230362s) [noise=true , address=104.36.82.47:4546 , piece_id=7JKKJPM3EJORWD5JWRJXLWU2XBOVORETDQ5B4NPKFAOCOAGF6ANA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: write tcp 159.223.143.39:56188->104.36.82.47:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294391262 (26h29m57.075324637s) [noise=true , address=104.36.82.13:4208 , piece_id=FIKJUW647V3WH762O22OUEHLCYFBUUPMPMVFI5P62IPSKZKAVEWA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12SXSN3HpeYwgHRpFFST5wwP3hhXEe3QB8VzWTNNE19sFoWW4o5, address:104.36.82.13:4208): protocol: write tcp 159.223.143.39:33180->104.36.82.13:4208: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294531549 (26h29m57.075464924s) [noise=true , address=104.36.82.46:4515 , piece_id=NP2EU4N3DIUMZOIWPCYCCBQZ5GHVQYKKSU4Z72GREJALGHZIGZCA, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:52480->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294588877 (26h29m57.075522255s) [noise=true , address=104.36.82.20:4444 , piece_id=C3JVJ35W6BAZ7CRRVUJZCJWWHXG3NSWQPHNKGRTO73QR755DNSEQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:4444): protocol: write tcp 159.223.143.39:44272->104.36.82.20:4444: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294635310 (26h29m57.075568687s) [noise=true , address=104.36.82.6:4906 , piece_id=QWM3NL6D3LIDGUII3QRWIOP5W7MGVRDQMRU7DWEGBSB2V4W22QBQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1gByS6jTUkLuHrY8MafgmkmB4wXyY2j9Mt48fkpPNWbCKTcnty, address:104.36.82.6:4906): protocol: write tcp 159.223.143.39:55650->104.36.82.6:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294710608 (26h29m57.07564399s) [noise=true , address=104.36.82.14:4227 , piece_id=JGP5VDKQ7DWJXO2D452CP5GAZZOERKP5TJHG7KQI5X6ND4VQOSOQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:104.36.82.14:4227): protocol: write tcp 159.223.143.39:38254->104.36.82.14:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294771859 (26h29m57.075705238s) [noise=true , address=104.36.82.4:4810 , piece_id=PZ7UD5VUMFVVKWSCDEQ7GLPUT6PWFVIKGLB4UBZCLVQEEXAQGOQQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:12GTWz2fWHLAApibeARUerDvJ9c4AWYRVAvumRxgJwwqCYYJJfg, address:104.36.82.4:4810): protocol: write tcp 159.223.143.39:43612->104.36.82.4:4810: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294798168 (26h29m57.075731536s) [noise=true , address=104.36.82.41:4793 , piece_id=3GGZZAHHV6MFSQM43SIS75MYAAZ5AFMMRHAGTXQOHP2VZJYCCAUQ, seg_pos={0 0}, upload=25273]: Done uploading piece. err: ecclient: upload failed (node:1asDxxBjiU6yNbZ5c6uBmSWQ4QbVs5J3ErLbpKBSwqaptpNhpn, address:104.36.82.41:4793): protocol: write tcp 159.223.143.39:34488->104.36.82.41:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:41.294844121 (26h29m57.075777527s) [seg_pos={0 0}, upload=25273]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:21:41.317380829 (26h29m57.098314199s) [upload=25273]: Done uploading segments... Jul 28 00:21:41.317406796 (26h29m57.098340175s) [upload=25273]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.025699891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X........X.....] duration: 463.804696ms FAIL (2025-07-28 00:21:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.844236ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X...XX.......] duration: 444.627472ms FAIL (2025-07-28 00:21:16 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 444.658704ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 295.437752ms FAIL (2025-07-28 00:20:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 295.464777ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X........X....] duration: 439.027244ms FAIL (2025-07-28 00:20:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.065316ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 206.807902ms FAIL (2025-07-28 00:20:42 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 206.825006ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [............................................XXX.XXXXXXXXXXXX] duration: 1m0.042735299s FAIL (2025-07-28 00:20:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 36.82.11:4120 , piece_id=22USLXSQ2PPCFT44WTDFAG67E7QZ5FSW4TMU4E6OP5KOLW5RBJQQ, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.377362204 (26h27m45.158295591s) [noise=true , address=104.36.82.15:4264 , piece_id=K7DH2DUQUYB7XMYHBM4ECOHNNA3BUFZS4AUUK6KYBNHX655GGZPQ, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.377680173 (26h27m45.15861355s) [noise=true , address=104.36.82.14:4239 , piece_id=L2G37GHJ2ECW6KD6OBGXDD25L3N5EV54JVIWJ2RIQAG3QP2NVJ2A, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.378003890 (26h27m45.158937288s) [noise=true , address=104.36.82.9:4016 , piece_id=52TFQCPW7B3XKFX5D2NRVTMSO7NA7X73B3P753NN7I3VDGK7AV6A, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.378386515 (26h27m45.159319903s) [noise=true , address=104.36.82.12:4158 , piece_id=G5GECNJTJNWDMQG5CJEHQOBPBJOTSODWSFAATWSOTSXV4TCPMW6Q, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.378785141 (26h27m45.159718525s) [noise=true , address=104.36.82.54:4190 , piece_id=UQXSHKCGENLC7TPOS55VACWVGFVHOHR4M47ID6KYVKECDLHCT63Q, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.381296526 (26h27m45.162229904s) [noise=true , address=104.36.82.49:4979 , piece_id=LVDT2ESGPWCATTLJV63I27I4PMIF7JRYCXOJJDH4K5VSAF25AB4A, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:29.381648816 (26h27m45.162582193s) [noise=true , address=104.36.82.46:4482 , piece_id=QN6VOZPTXO2IGILZEMCU47R53Z5ERFSABMQ4PNYSLD2UAS3XDHWQ, seg_pos={0 0}, upload=25254]: Uploading piece... Jul 28 00:19:32.439087572 (26h27m48.22002095s) [upload=25254]: Next returned nil segment Jul 28 00:19:32.439108645 (26h27m48.220042029s) [upload=25254]: Waiting for error group managing segments... Jul 28 00:19:37.374047758 (26h27m53.154981123s) [noise=true , address=64.49.13.154:4932 , piece_id=N6TJVE5VR6XRZ6JHNTTND6CSM7SKWZWODMW3KSJ222QPE2WUCOSQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: <nil> Jul 28 00:20:29.280996877 (26h28m45.061930263s) [noise=true , address=104.36.82.49:4979 , piece_id=LVDT2ESGPWCATTLJV63I27I4PMIF7JRYCXOJJDH4K5VSAF25AB4A, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: write tcp 159.223.143.39:54428->104.36.82.49:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281092777 (26h28m45.062026153s) [noise=true , address=104.36.82.15:4264 , piece_id=K7DH2DUQUYB7XMYHBM4ECOHNNA3BUFZS4AUUK6KYBNHX655GGZPQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1ovYsPwYdx7AYPboFx3Xdcg4VQrPGZj2XTMTqgox6zm81gbf8Z, address:104.36.82.15:4264): protocol: rpc: dial tcp 104.36.82.15:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281156071 (26h28m45.062089446s) [noise=true , address=104.36.82.47:4531 , piece_id=UG4ZPG22I3TOBQFKQF6DOIAXZP2PHT6JIPZMOJMQIANF3FWMKMJA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281214921 (26h28m45.062148302s) [noise=true , address=104.36.82.16:4303 , piece_id=4WPMNU26H35OFD7BHDAJHX276V6VGNBQPBWZ2CN3K63L2PVOW2LA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12Ec4Xvp3nGRAm9BivLLTp6AVkMoz1i2AgXQdfQaAmNPLHWD4Yv, address:104.36.82.16:4303): protocol: rpc: dial tcp 104.36.82.16:4303: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281239724 (26h28m45.062173089s) [noise=true , address=104.36.82.42:4830 , piece_id=2KTKBFJVFHCEJJJBFA3JZBUKO3IQAXUIV26WK4NBTZNLTXSI6JPA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: write tcp 159.223.143.39:47004->104.36.82.42:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281283966 (26h28m45.062217342s) [noise=true , address=104.36.82.47:4538 , piece_id=77P2B7JGYLRIWBA2FQKOFXR2JM6EVVZSTB27MZRNG4K3K2DMUSVQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: rpc: dial tcp 104.36.82.47:4538: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281332944 (26h28m45.062266309s) [noise=true , address=104.36.82.13:4195 , piece_id=EMQ5AZTCK7TDNTPFE3SH34SSINCU2SYJEYLEX2TINND5H35RBKOA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12mmoBC7kBL5PVuzYvsq8Qb2kFNJNWQBTSCVq6xjkeyJxXPdPBH, address:104.36.82.13:4195): protocol: rpc: dial tcp 104.36.82.13:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281350402 (26h28m45.062283781s) [noise=true , address=104.36.82.10:4101 , piece_id=757XX3FGCBWBHYTVHOA3NEKW5LIBQ2ZHANEFFBSJQBCXHXUNTJWA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281403116 (26h28m45.062336479s) [noise=true , address=104.36.82.11:4120 , piece_id=22USLXSQ2PPCFT44WTDFAG67E7QZ5FSW4TMU4E6OP5KOLW5RBJQQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:123jcA328VCSUaWMKvSaeXip1U6Sc4SSvuRaoHF8fvsF6Tt6DHW, address:104.36.82.11:4120): protocol: rpc: dial tcp 104.36.82.11:4120: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281430975 (26h28m45.062364353s) [noise=true , address=104.36.82.14:4239 , piece_id=L2G37GHJ2ECW6KD6OBGXDD25L3N5EV54JVIWJ2RIQAG3QP2NVJ2A, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:104.36.82.14:4239): protocol: rpc: dial tcp 104.36.82.14:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281472144 (26h28m45.062405508s) [noise=true , address=104.36.82.56:4256 , piece_id=R7FOOWRK5HM3W6UAVRS7MDII5M5JLXTCOSUXSNIT52ORF4NR6WPA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281495091 (26h28m45.062428473s) [noise=true , address=104.36.82.9:4016 , piece_id=52TFQCPW7B3XKFX5D2NRVTMSO7NA7X73B3P753NN7I3VDGK7AV6A, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:104.36.82.9:4016): protocol: rpc: dial tcp 104.36.82.9:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281508444 (26h28m45.062441815s) [noise=true , address=104.36.82.46:4482 , piece_id=QN6VOZPTXO2IGILZEMCU47R53Z5ERFSABMQ4PNYSLD2UAS3XDHWQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281520623 (26h28m45.062454004s) [noise=true , address=104.36.82.4:4828 , piece_id=PUQMOE3ZPGLXJPATIWP7ECKUI2BGUBD5HPFDHND3PNZIPPZXP7HQ, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: rpc: dial tcp 104.36.82.4:4828: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281558796 (26h28m45.062492158s) [noise=true , address=104.36.82.12:4158 , piece_id=G5GECNJTJNWDMQG5CJEHQOBPBJOTSODWSFAATWSOTSXV4TCPMW6Q, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:4158): protocol: rpc: dial tcp 104.36.82.12:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281573549 (26h28m45.062506911s) [noise=true , address=104.36.82.54:4190 , piece_id=UQXSHKCGENLC7TPOS55VACWVGFVHOHR4M47ID6KYVKECDLHCT63Q, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: rpc: dial tcp 104.36.82.54:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281609225 (26h28m45.062542603s) [noise=true , address=104.36.82.44:4886 , piece_id=IVWTKZXQDR7ZYFZ3MBVQ322HWYNHDKYJU2OHRCR6OO4OHGEP63TA, seg_pos={0 0}, upload=25254]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 159.223.143.39:40152->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:29.281653022 (26h28m45.062586401s) [seg_pos={0 0}, upload=25254]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:20:29.320135632 (26h28m45.101068997s) [upload=25254]: Done uploading segments... Jul 28 00:20:29.320335930 (26h28m45.101269316s) [upload=25254]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.042757999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X...XX......] duration: 517.970194ms FAIL (2025-07-28 00:20:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.014438ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X...XX.....] duration: 503.919258ms FAIL (2025-07-28 00:19:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 503.963192ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X........X...] duration: 450.965282ms FAIL (2025-07-28 00:19:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 451.01738ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.............................................XXX.XXXXXXXXXXX] duration: 1m0.039265787s FAIL (2025-07-28 00:19:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== {0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.273885902 (26h26m45.054819285s) [noise=true , address=104.36.82.15:4289 , piece_id=5CDF7LTVTBSRYQAXBOTCNK7DPIU4QKH6VMDXQMT4VVJSZJ5XGO7Q, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.274337690 (26h26m45.055271073s) [noise=true , address=79.127.219.47:4059 , piece_id=VAMH2NLZ24ETD4323KIBCFWQXCVO6VK5YEFP464NYYXAL6HIJU4Q, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.274692246 (26h26m45.055625626s) [noise=true , address=104.36.82.21:4507 , piece_id=LJDZJQYEVRJ33BU7B4C2KYKZP2653RGRBBUCSFZSUMHKGHXF5KFQ, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.275892757 (26h26m45.056826126s) [noise=true , address=104.36.82.52:4120 , piece_id=RPT6KP4NMHWDIJHA4EAWIMFDMXNQWI3HBBV7POLJIGWLG5MFIVRA, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.276541496 (26h26m45.057474884s) [noise=true , address=104.36.82.12:4180 , piece_id=TLBF53U3UFRTUJYQOMC3JCXD4AJ2ECGMKBWQQFXVDD35SY63P7TA, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.277637482 (26h26m45.058570864s) [noise=true , address=104.36.82.52:4118 , piece_id=RXWTAUAYQ73NNIBJHORCXGQD3UNPVH3XSMOECIVFXJJRZ2AED5EQ, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.278764217 (26h26m45.059697599s) [noise=true , address=104.36.82.56:4251 , piece_id=VZYHKR7GHCF6A6S73E66Q2MRSDIUJTWL6P73PSE7LZ7D52Z2HBTQ, seg_pos={0 0}, upload=25239]: Uploading piece... Jul 28 00:18:29.402131013 (26h26m45.183064391s) [upload=25239]: Next returned nil segment Jul 28 00:18:29.402153801 (26h26m45.183087185s) [upload=25239]: Waiting for error group managing segments... Jul 28 00:18:29.417138111 (26h26m45.198071481s) [noise=true , address=79.127.219.47:4059 , piece_id=VAMH2NLZ24ETD4323KIBCFWQXCVO6VK5YEFP464NYYXAL6HIJU4Q, seg_pos={0 0}, upload=25239]: Done uploading piece. err: <nil> Jul 28 00:18:35.185318275 (26h26m50.966251642s) [noise=true , address=64.49.13.138:4482 , piece_id=KVFWNESH47XW4O6RI6KOTSLLGP6NKPXAX5PTPBNMDFNWMOYWLREQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: <nil> Jul 28 00:19:29.172966257 (26h27m44.953899635s) [noise=true , address=104.36.82.55:4201 , piece_id=POKAYQD3BN3TTDF3NZVIJ4BUQTAT4ICELNXNRFUYX2PGT6CQZFXQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173062301 (26h27m44.953995678s) [noise=true , address=104.36.82.51:4052 , piece_id=RG5TEDO22WUVAGZ3YEJ6XLNS2RNBIIB3KCBB4NYJBW3UP73KEFCA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12PViJBYAanwXRWhMq3WHR6k9Stxy8hjuY5FV9J7ErxCy4N1q3V, address:104.36.82.51:4052): protocol: write tcp 159.223.143.39:42914->104.36.82.51:4052: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173119991 (26h27m44.954053367s) [noise=true , address=104.36.82.53:4147 , piece_id=UCL5K2NJFXNUODVY76MXCQLDZ3FT4ADMMB4N62LVWMS3QMKVMX3A, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: rpc: dial tcp 104.36.82.53:4147: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173179932 (26h27m44.954113311s) [noise=true , address=104.36.82.40:4954 , piece_id=P3ES5RQVFAJTKBX42ITWNPGEIOHKXGE42YWUJEAYWDL7TUQ7IRJA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:14F5pJPTejP7BMpo1tcZo6uG9XdxvT7gLAUb1cgeBzKChafSXz, address:104.36.82.40:4954): protocol: rpc: dial tcp 104.36.82.40:4954: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173261077 (26h27m44.954194454s) [noise=true , address=104.36.82.15:4289 , piece_id=5CDF7LTVTBSRYQAXBOTCNK7DPIU4QKH6VMDXQMT4VVJSZJ5XGO7Q, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:126WPF6MtX7YjuKf73rGJrifdXyd78KY4FdL757qDipGNHBW6uY, address:104.36.82.15:4289): protocol: rpc: dial tcp 104.36.82.15:4289: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173320699 (26h27m44.954254076s) [noise=true , address=104.36.82.11:4119 , piece_id=GM7O74VBJAYA4EWWOOXXPCRUJELNE5ZWZBEJSZJGSCCKXL4F3ZLA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:121xNAnkm9HZXUszp26UmWTsuhp3igyeDVea6gSCdtrQUQDVHpW, address:104.36.82.11:4119): protocol: write tcp 159.223.143.39:49534->104.36.82.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173477079 (26h27m44.95441046s) [noise=true , address=104.36.82.52:4120 , piece_id=RPT6KP4NMHWDIJHA4EAWIMFDMXNQWI3HBBV7POLJIGWLG5MFIVRA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 159.223.143.39:53522->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173575407 (26h27m44.954508774s) [noise=true , address=104.36.82.56:4251 , piece_id=VZYHKR7GHCF6A6S73E66Q2MRSDIUJTWL6P73PSE7LZ7D52Z2HBTQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: write tcp 159.223.143.39:37108->104.36.82.56:4251: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173657398 (26h27m44.954590776s) [noise=true , address=104.36.82.10:4089 , piece_id=J4677GYQ37VV4K2PKOEKOZJOQFC6S3FHYMSVM4MNWFPNB5NP2YGQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 159.223.143.39:56924->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173732877 (26h27m44.954666257s) [noise=true , address=104.36.82.12:4180 , piece_id=TLBF53U3UFRTUJYQOMC3JCXD4AJ2ECGMKBWQQFXVDD35SY63P7TA, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1ky8YcXCwXBhwHMFA2Rj6wPetB3bkpmnL5X8XJLqXgLvZuu3Kt, address:104.36.82.12:4180): protocol: write tcp 159.223.143.39:51846->104.36.82.12:4180: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173793280 (26h27m44.95472665s) [noise=true , address=104.36.82.41:4773 , piece_id=4XHKBVBTZURZ2Z2HHFKKFX6LIMSP45BJO3FPIWBVZVX243FSGH3A, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV, address:104.36.82.41:4773): protocol: rpc: dial tcp 104.36.82.41:4773: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173881723 (26h27m44.954815102s) [noise=true , address=104.36.82.20:4438 , piece_id=5VVBGIGUKEVOFU3UVK2HZZZGVXNDPPDCFKG56Z3PYUMTHNFVKBSQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:104.36.82.20:4438): protocol: write tcp 159.223.143.39:41580->104.36.82.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173906616 (26h27m44.954839983s) [noise=true , address=104.36.82.52:4118 , piece_id=RXWTAUAYQ73NNIBJHORCXGQD3UNPVH3XSMOECIVFXJJRZ2AED5EQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:104.36.82.52:4118): protocol: write tcp 159.223.143.39:56602->104.36.82.52:4118: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173955869 (26h27m44.95488931s) [noise=true , address=104.36.82.13:4207 , piece_id=RBCSX7J254AYNU2X7XXLKPKRTMAGUGVFCDBPKNMIC3VLO3AY342A, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: write tcp 159.223.143.39:53260->104.36.82.13:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.173981848 (26h27m44.954915214s) [noise=true , address=104.36.82.21:4507 , piece_id=LJDZJQYEVRJ33BU7B4C2KYKZP2653RGRBBUCSFZSUMHKGHXF5KFQ, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: write tcp 159.223.143.39:53224->104.36.82.21:4507: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.174015468 (26h27m44.95494883s) [noise=true , address=104.36.82.47:4525 , piece_id=LDYLIHZDICA2XMMODEEOU7QJFNK5LZZNNDN54KJRQ77MAA3UB22Q, seg_pos={0 0}, upload=25239]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: write tcp 159.223.143.39:42642->104.36.82.47:4525: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:29.174055890 (26h27m44.954989258s) [seg_pos={0 0}, upload=25239]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:19:29.209878828 (26h27m44.990812205s) [upload=25239]: Done uploading segments... Jul 28 00:19:29.209930952 (26h27m44.990864337s) [upload=25239]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.039286649s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 254.597819ms FAIL (2025-07-28 00:19:29 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 254.634245ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X...XX....] duration: 364.104045ms FAIL (2025-07-28 00:19:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 364.14642ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X........X..] duration: 528.847256ms FAIL (2025-07-28 00:18:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 528.901787ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X........X.] duration: 460.855432ms FAIL (2025-07-28 00:18:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 461.30192ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X........X] duration: 30.030041473s FAIL (2025-07-28 00:18:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== VNCOWB4AFOXI5T7I5NLBVIRDXPUWUY6PQND4XJ46NU7Q, seg_pos={0 0}, upload=25231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT): protocol: rpc: dial tcp 104.36.82.16:4295: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.220424821 (26h26m11.001358205s) [noise=true , address=77.90.31.21:4879 , piece_id=ZTKBYDDPQLQTNCUES5BHEPGGE3G2L5XCZHQWT22QSR3HAYBIYNFA, seg_pos={0 0}, upload=25231]: Done uploading piece. err: <nil> Jul 28 00:17:55.220464365 (26h26m11.001397769s) [seg_pos={0 0}, upload=25231]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Jul 28 00:17:55.220560135 (26h26m11.001493513s) [upload=25231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:17:55.220567172 (26h26m11.001500572s) [upload=25231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:17:55.271837722 (26h26m11.052771116s) [upload=25231]: Done uploading segments... Jul 28 00:17:55.271868668 (26h26m11.052802037s) [upload=25231]: Upload finished. err: <nil> Jul 28 00:18:12.763418813 (26h26m28.54435218s) [upload=25235]: Starting upload Jul 28 00:18:12.763590227 (26h26m28.544523618s) [upload=25235]: Done starting upload Jul 28 00:18:12.763729384 (26h26m28.544662763s) [upload=25235]: Uploading segments... Jul 28 00:18:12.764894309 (26h26m28.54582771s) [upload=25235]: Got next segment. Inline: false Jul 28 00:18:12.764974525 (26h26m28.545907909s) [upload=25235]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:18:12.764982233 (26h26m28.545915622s) [upload=25235]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:18:12.796137606 (26h26m28.577070995s) [seg_pos={0 0}, upload=25235]: Begin upload segment... Jul 28 00:18:12.796346006 (26h26m28.577279375s) [seg_pos={0 0}, upload=25235]: Done begin upload segment. Jul 28 00:18:12.796626998 (26h26m28.577560377s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.796692365 (26h26m28.577625751s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.797743277 (26h26m28.578676646s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798409285 (26h26m28.579342654s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.798748953 (26h26m28.579682323s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799370166 (26h26m28.580303536s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.799970676 (26h26m28.580904046s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.801451228 (26h26m28.582384603s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.802599527 (26h26m28.583532909s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.803369736 (26h26m28.584303107s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.804132236 (26h26m28.585065606s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.805007070 (26h26m28.585940441s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.806990441 (26h26m28.587923811s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.810757571 (26h26m28.591690953s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.814292397 (26h26m28.595225781s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.818477534 (26h26m28.599410897s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Uploading piece... Jul 28 00:18:12.941646067 (26h26m28.722579455s) [upload=25235]: Next returned nil segment Jul 28 00:18:12.941666609 (26h26m28.7226s) [upload=25235]: Waiting for error group managing segments... Jul 28 00:18:12.950815116 (26h26m28.731748561s) [noise=true , address=77.90.31.11:4554 , piece_id=SU2UCTBOOMU7IC5UXCGUHJVZXLT6GV27C5US6ESRGTTLR5FLRHZQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.950877185 (26h26m28.731810548s) [noise=true , address=77.90.31.21:4879 , piece_id=CH2WM2KNRKEV75JFJ7ALFQT65XW7M3KVACU3W4PVSLKFKJOQ2K3Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951669562 (26h26m28.732602925s) [noise=true , address=45.91.116.11:4117 , piece_id=TKEDXADFUKAF2E6E5OSTCPV2TRR3BI4OKJW3IVME66DNEARQ3Z5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.951728641 (26h26m28.732662017s) [noise=true , address=45.91.116.12:4225 , piece_id=OIDIK3LFSZ77MX4ABNNFUQD73ANQEDJKH7CLMTK4AORN7BGUS2HQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952460467 (26h26m28.733393836s) [noise=true , address=77.90.31.12:4589 , piece_id=5YTMNK3CAFELU5SYNIDBQBR3ZHKCAISTNKIWZSFIX2CVB2YBA4BA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.952617617 (26h26m28.733550997s) [noise=true , address=77.90.31.18:4805 , piece_id=O5LCJEJAGJT2KNXHJPYW6HFRCXAQYWUMZTW7QJ3RNMJGYF7G34VQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.953322257 (26h26m28.734255624s) [noise=true , address=77.90.31.12:4592 , piece_id=CMPKGBI4PXFU4FGRKN4CS5PVG5TSZ7VRVY6WMP3PFV4DTVOBPPVQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954092305 (26h26m28.735025685s) [noise=true , address=45.91.116.11:4118 , piece_id=D3CPS2EJWJJOSHSCU2EPMPG4QYJBSIKQCNZQGKLZRNW5QKWFZJAQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954512638 (26h26m28.735446004s) [noise=true , address=45.91.116.14:4191 , piece_id=2TILVONSLFYU3PFFBVALNIIFISC4EYZTPVO6BLUVP7XT5GW2TQ2Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.954839265 (26h26m28.735772646s) [noise=true , address=77.90.31.14:4663 , piece_id=HUERG6JENY4M5LTMYFH2ZJACEZICFWT4AMT3YLBSZO3IQV3JBJ5Q, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.959271938 (26h26m28.740205315s) [noise=true , address=77.90.31.16:4733 , piece_id=SG2RSYPL72J4MAMEFRP224EAOH2AVFBARXJAQX6CJXTMJSDZI4XA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:12.965948053 (26h26m28.746881417s) [noise=true , address=77.90.31.16:4735 , piece_id=IF4XYUP3HDHWSJFTKS7ZVDWEBJOQUGZ64TIRWNOCOFPNVJT6VFYQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: <nil> Jul 28 00:18:42.722625290 (26h26m58.503558665s) [noise=true , address=104.36.82.2:4730 , piece_id=U243UA23ZGJXUHRIG2BW3ONF3VIEABRJVG3NILIZXXKLR7RG42GQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722753648 (26h26m58.503687015s) [noise=true , address=104.36.82.6:4873 , piece_id=YGQEQ6ZEG6QOPHJ2HWQGJP3ROBPIY3EARTO7BJYPEJ6RATMTH2JQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722822371 (26h26m58.503755737s) [noise=true , address=104.36.82.9:4006 , piece_id=YAJQ2U5KJ7AXWUZ7THEL2F32TTBXPR2E3XJOB3234SGHBCKJ6GLQ, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:42.722918460 (26h26m58.503851837s) [noise=true , address=104.36.82.7:4910 , piece_id=BOIM23BCWQZFQW3B5UCWVAUNWTOH4RMZLU5C3MYMGAA4GINWUIZA, seg_pos={0 0}, upload=25235]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:18:42.722967264 (26h26m58.50390065s) [seg_pos={0 0}, upload=25235]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:18:42.751058707 (26h26m58.531992075s) [upload=25235]: Done uploading segments... Jul 28 00:18:42.751082224 (26h26m58.532015599s) [upload=25235]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.030070389s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X...XX...] duration: 437.944227ms FAIL (2025-07-28 00:18:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.969655ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..............................................XXX.XXXXXXXXXX] duration: 1m0.038180749s FAIL (2025-07-28 00:18:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== Z6WQ, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.176161354 (26h25m44.957094739s) [noise=true , address=104.36.82.56:4231 , piece_id=O2S62AEVWPRCU45QFTL67K5M4H6ZCGFFBA2FDP5Z6UZHTUZ6BNZA, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.176690813 (26h25m44.957624201s) [noise=true , address=104.36.82.40:4961 , piece_id=TGQXJJ3RINAEMIBNMULXX2P25ARQC2HJZAZGA72I62DQHIN32QMQ, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.177114170 (26h25m44.958047558s) [noise=true , address=104.36.82.42:4810 , piece_id=MBT3EKLSBYBICYOJZ4LGF5APPLWV2JYGFCOMMIXBLIINM7T5VAZQ, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.177557861 (26h25m44.958491251s) [noise=true , address=104.36.82.12:4183 , piece_id=OLGVJPJVPICNOVNSHPTGP5NR4S5HRAIRP727HH64NDNWW2QBYHBA, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.177956445 (26h25m44.958889833s) [noise=true , address=104.36.82.41:4783 , piece_id=OUQTNKJKGILHYNFX2OJUQOUNC677RZMXARYAPZNKEB6EDXBJH6XA, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.178373228 (26h25m44.959306616s) [noise=true , address=104.36.82.53:4136 , piece_id=GAIUVB7JL7IGEQTX6YLY3DDUTU2FIDI6T4WIDPGCXOU6RG7PHCOA, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.178843365 (26h25m44.959776751s) [noise=true , address=104.36.82.53:4150 , piece_id=ODBMXXBA2J6GT27IQDTDHBF36OYOKUGN4MVKYE2VINDB47XXB5PQ, seg_pos={0 0}, upload=25222]: Uploading piece... Jul 28 00:17:29.740846651 (26h25m45.521780017s) [upload=25222]: Next returned nil segment Jul 28 00:17:29.740864506 (26h25m45.521797893s) [upload=25222]: Waiting for error group managing segments... Jul 28 00:17:30.443334967 (26h25m46.224268336s) [noise=true , address=64.49.13.140:4508 , piece_id=FYWUZKAC3K2LLYHBGYT4LRHCDYWXAPNWBBDD3CGQPRBKC3WJ4TYA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: <nil> Jul 28 00:18:29.060741558 (26h26m44.841674941s) [noise=true , address=104.36.82.52:4116 , piece_id=HAQG6WWZ2LZWS7N575I4667ZKYQHXWJVBIGPV2Y7BA4PXZZAGMVA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12vKCXgWTAsipTvUK9XA3sL6DznFA8BqZc6WCN84PBuLhKCTT9m, address:104.36.82.52:4116): protocol: rpc: dial tcp 104.36.82.52:4116: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.060853783 (26h26m44.841787163s) [noise=true , address=104.36.82.40:4961 , piece_id=TGQXJJ3RINAEMIBNMULXX2P25ARQC2HJZAZGA72I62DQHIN32QMQ, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12ZGPLhM6NsdjKLkLCCLSeaNS8ikfnHPZxc3CCw9mLzSznbEoUa, address:104.36.82.40:4961): protocol: rpc: dial tcp 104.36.82.40:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.060944766 (26h26m44.841878143s) [noise=true , address=104.36.82.53:4150 , piece_id=ODBMXXBA2J6GT27IQDTDHBF36OYOKUGN4MVKYE2VINDB47XXB5PQ, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: rpc: dial tcp 104.36.82.53:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061037767 (26h26m44.841971148s) [noise=true , address=104.36.82.10:4100 , piece_id=2QI7YDJ2BBTM756UB37PDDZTZ2XRL2OOBHHVYK3FBJ25CO3C6MWA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12Hyc6sbtuPXMLyFzubMcepbJyLyVAGNpkXAmqg73i3sY97dwdW, address:104.36.82.10:4100): protocol: rpc: dial tcp 104.36.82.10:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061124833 (26h26m44.842058198s) [noise=true , address=104.36.82.11:4124 , piece_id=WT6YQAOHWHPQLMWUXEUO24WSST6TC5O6VNF4BEKRC5LREWKK254A, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1QAgfinEkxYwjzvtr53FRritkoDNbfYAw21qcJuj3uCY4JT4cZ, address:104.36.82.11:4124): protocol: write tcp 159.223.143.39:36878->104.36.82.11:4124: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061256864 (26h26m44.842190241s) [noise=true , address=104.36.82.54:4177 , piece_id=2V53I2BQTBAB26W4KUR3MUV3V3ZFZOSFJ5FJSQVV2TYOFE4SZ6WQ, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: rpc: dial tcp 104.36.82.54:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061348016 (26h26m44.842281383s) [noise=true , address=104.36.82.20:4440 , piece_id=TXXXSXEYYVVOYOVXSBGDJHZFP4SEVUJPN5RVN4QZRZL4E7RH77SA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:104.36.82.20:4440): protocol: write tcp 159.223.143.39:48254->104.36.82.20:4440: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061540322 (26h26m44.842473712s) [noise=true , address=104.36.82.41:4783 , piece_id=OUQTNKJKGILHYNFX2OJUQOUNC677RZMXARYAPZNKEB6EDXBJH6XA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1sNMd8YrgaagKDoWivhbTmRc5TPM6BNiHscECBF2yhncKgkvz, address:104.36.82.41:4783): protocol: rpc: dial tcp 104.36.82.41:4783: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.061674490 (26h26m44.842607874s) [noise=true , address=104.36.82.53:4136 , piece_id=GAIUVB7JL7IGEQTX6YLY3DDUTU2FIDI6T4WIDPGCXOU6RG7PHCOA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:104.36.82.53:4136): protocol: rpc: dial tcp 104.36.82.53:4136: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062027930 (26h26m44.842961317s) [noise=true , address=104.36.82.5:4868 , piece_id=G5WRV32ZU2EIRR3XB35WIKGARUJW5J43IX3PWQRDLDZVJNC4CFWA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:18Srfics6dFkQeHZ2WF5YkFuLm2719Pw4Xgg9ntk12KnTBcn8v, address:104.36.82.5:4868): protocol: rpc: dial tcp 104.36.82.5:4868: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062182107 (26h26m44.843115493s) [noise=true , address=104.36.82.54:4171 , piece_id=6TKFX7EEBR7DS3V6IVVYW6EGJGZ2UUBZ6IEI5JAUL33OVHF4GVZA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: write tcp 159.223.143.39:39942->104.36.82.54:4171: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062276020 (26h26m44.843209408s) [noise=true , address=104.36.82.15:4272 , piece_id=MRSZRGCPKROGT3S7W7NZELOLV7NYEWZRJV2QIF7L3HPDIHRXJQOA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:104.36.82.15:4272): protocol: rpc: dial tcp 104.36.82.15:4272: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062527546 (26h26m44.843460943s) [noise=true , address=104.36.82.55:4218 , piece_id=6SN7STKHR3KZHPXXSW4PZS6CD7GL6OXDG6TMXM4I7XZEQXJK6GWA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: write tcp 159.223.143.39:36342->104.36.82.55:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062654579 (26h26m44.84358796s) [noise=true , address=104.36.82.42:4810 , piece_id=MBT3EKLSBYBICYOJZ4LGF5APPLWV2JYGFCOMMIXBLIINM7T5VAZQ, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1CGnNCsXi1RAnTx9P28561Q1wK6LBqpBzqyuowxaAZJETQRz2w, address:104.36.82.42:4810): protocol: rpc: dial tcp 104.36.82.42:4810: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062689601 (26h26m44.843622984s) [noise=true , address=104.36.82.12:4183 , piece_id=OLGVJPJVPICNOVNSHPTGP5NR4S5HRAIRP727HH64NDNWW2QBYHBA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: rpc: dial tcp 104.36.82.12:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.062748667 (26h26m44.843682038s) [noise=true , address=104.36.82.6:4906 , piece_id=SH7JWFEATCGVT74VT2BBWQGPF745AJRLXX2FRUVWBCXAIBJVPEOA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:1gByS6jTUkLuHrY8MafgmkmB4wXyY2j9Mt48fkpPNWbCKTcnty, address:104.36.82.6:4906): protocol: write tcp 159.223.143.39:35234->104.36.82.6:4906: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.063006548 (26h26m44.843939939s) [noise=true , address=104.36.82.56:4231 , piece_id=O2S62AEVWPRCU45QFTL67K5M4H6ZCGFFBA2FDP5Z6UZHTUZ6BNZA, seg_pos={0 0}, upload=25222]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: rpc: dial tcp 104.36.82.56:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:29.063068946 (26h26m44.844002341s) [seg_pos={0 0}, upload=25222]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:18:29.094631334 (26h26m44.87556471s) [upload=25222]: Done uploading segments... Jul 28 00:18:29.094689068 (26h26m44.875622436s) [upload=25222]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.038207483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 272.974794ms FAIL (2025-07-28 00:18:26 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 273.026998ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 212.242612ms FAIL (2025-07-28 00:18:06 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 212.273926ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 447.086986ms FAIL (2025-07-28 00:17:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.120271ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X...XX..] duration: 546.318683ms FAIL (2025-07-28 00:17:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 546.352273ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 497.878573ms FAIL (2025-07-28 00:17:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 497.913512ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [...............................................XXX.XXXXXXXXX] duration: 1m0.036045786s FAIL (2025-07-28 00:17:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== g piece... Jul 28 00:16:27.862858546 (26h24m43.64379193s) [noise=true , address=77.90.31.10:4550 , piece_id=5NVINMEPBIBWZSDMPONBJQYRAC6DKUTMK6QAFDVQK5FFGGUGKVVQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863013246 (26h24m43.643946628s) [noise=true , address=104.36.82.5:4856 , piece_id=5ERJ27Z546BOTRAQWJNPRFXWS2FWHNODVEGMNROJNSB5UPUPXSBQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863316458 (26h24m43.644249846s) [noise=true , address=104.36.82.12:4184 , piece_id=GPDRBM723G7SN4QCFGUN6X4DLWVL6Y3EWH3VSG53X4RE5ILXVUGA, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863437144 (26h24m43.644370528s) [noise=true , address=104.36.82.14:4237 , piece_id=DTU64USY5MGTBQA2DAQZAB22OWWXIS3JU53SH2LSCYXDIZMPPHBQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863703523 (26h24m43.6446369s) [noise=true , address=104.36.82.15:4279 , piece_id=SXEKQF4XDV2ZEIWFRTZAS7ZIJC6UQXJKRXOQTTNW57LC77SNO7CQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.863819189 (26h24m43.644752589s) [noise=true , address=79.127.201.215:4350 , piece_id=TVQ3Y6A2YCQB73DH6KGGTSF3MB7FY4DS6ATAHJRCOPAGN3GQDIGA, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.864070622 (26h24m43.645004001s) [noise=true , address=104.36.82.11:4138 , piece_id=D7S2HKPPXKVHEK2CEPB2VYOQQUCZ4HOUJKBIYDC7LAIAYBMSXUDQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.864321390 (26h24m43.645254777s) [noise=true , address=104.36.82.56:4251 , piece_id=PGVE5V7KGECIXX4CR5BUQGQP3UVJ73HFID6MI7GOAIEOAOEO5LTQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.864599193 (26h24m43.64553257s) [noise=true , address=104.36.82.52:4120 , piece_id=N2WBQD4TF5MYSCK6OW7DDU45S6VWBRCFR5VHUIRYKWLAGLOBT3QQ, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:27.864831161 (26h24m43.645764549s) [noise=true , address=104.36.82.16:4301 , piece_id=TYJHHPC45N5ABFEBZOKW7BYHIKW76TLGEWP6XMH3T4TO47VS3SBA, seg_pos={0 0}, upload=25207]: Uploading piece... Jul 28 00:16:28.022063143 (26h24m43.80299651s) [upload=25207]: Next returned nil segment Jul 28 00:16:28.022094667 (26h24m43.803028044s) [upload=25207]: Waiting for error group managing segments... Jul 28 00:16:28.183685599 (26h24m43.964618967s) [noise=true , address=77.90.31.10:4550 , piece_id=5NVINMEPBIBWZSDMPONBJQYRAC6DKUTMK6QAFDVQK5FFGGUGKVVQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: <nil> Jul 28 00:16:28.505796384 (26h24m44.286729756s) [noise=true , address=79.127.205.234:4613 , piece_id=LA2TID24XTDHJU6GHRAAZBOBWZYOTU7YIGGK2XNB6SWYFKFYU6IA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: <nil> Jul 28 00:16:35.139980295 (26h24m50.92091368s) [noise=true , address=79.127.201.215:4350 , piece_id=TVQ3Y6A2YCQB73DH6KGGTSF3MB7FY4DS6ATAHJRCOPAGN3GQDIGA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: <nil> Jul 28 00:17:27.784503680 (26h25m43.565437066s) [noise=true , address=104.36.82.44:4894 , piece_id=365MWDHILME2ETIEEJQLCHEA6W4RHRRIZXVA44LAE72M4YBZQ4XA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: rpc: dial tcp 104.36.82.44:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784581479 (26h25m43.56551485s) [noise=true , address=104.36.82.15:4279 , piece_id=SXEKQF4XDV2ZEIWFRTZAS7ZIJC6UQXJKRXOQTTNW57LC77SNO7CQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784720082 (26h25m43.565653446s) [noise=true , address=104.36.82.10:4106 , piece_id=LCXM4L2RAAZXTMKT5RRCQKSPXMZMVIN7OM2BZ4HPNRFY5FWPL5OQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12CsQq2BuV9hNc5fVAsw882NkDJMEXZeNwhV2FGNGKAU6wuFBHm, address:104.36.82.10:4106): protocol: rpc: dial tcp 104.36.82.10:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784829846 (26h25m43.565763213s) [noise=true , address=104.36.82.11:4138 , piece_id=D7S2HKPPXKVHEK2CEPB2VYOQQUCZ4HOUJKBIYDC7LAIAYBMSXUDQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1H28XSTvFGa3XhavfhythHWs95cqjazu2RuqpsjNCmdSFvRLoh, address:104.36.82.11:4138): protocol: write tcp 159.223.143.39:34080->104.36.82.11:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784890243 (26h25m43.565823611s) [noise=true , address=104.36.82.56:4251 , piece_id=PGVE5V7KGECIXX4CR5BUQGQP3UVJ73HFID6MI7GOAIEOAOEO5LTQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.784952274 (26h25m43.565885656s) [noise=true , address=104.36.82.42:4806 , piece_id=4KNOPIDDZBCPWIOQJJXLNFYDB7GQNUBAZHVXZE4EXUMDKQ5IGYRA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1tWnxuTSo4yHAJUzvVFTmwPSNb8cBJJJ17BnKd5Z3HMYf66vSC, address:104.36.82.42:4806): protocol: rpc: dial tcp 104.36.82.42:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.785769660 (26h25m43.56670303s) [noise=true , address=104.36.82.53:4149 , piece_id=ONCUUK6NXATX4MDZT4KJAVRUCLMHBRLVXFC7UFAAWPAZ7HMZC2GQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1nDCYjyLUqt1zkfotDNVcor73Cdpq3fnyMiygujdc9V4tGNHLY, address:104.36.82.53:4149): protocol: rpc: dial tcp 104.36.82.53:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.785839624 (26h25m43.566773s) [noise=true , address=104.36.82.5:4856 , piece_id=5ERJ27Z546BOTRAQWJNPRFXWS2FWHNODVEGMNROJNSB5UPUPXSBQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12mXTGa9rxx8jU8oKLmR9obX3Utn7p5rQH2GeK6LeJ6jiNvfYZ6, address:104.36.82.5:4856): protocol: rpc: dial tcp 104.36.82.5:4856: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.785903354 (26h25m43.566836735s) [noise=true , address=104.36.82.14:4237 , piece_id=DTU64USY5MGTBQA2DAQZAB22OWWXIS3JU53SH2LSCYXDIZMPPHBQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:104.36.82.14:4237): protocol: rpc: dial tcp 104.36.82.14:4237: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.785977411 (26h25m43.566910796s) [noise=true , address=104.36.82.9:4016 , piece_id=7FCSE65GL7OZB3P36TF4FL44BV4BIECGABLPAA7XFABQD35UGNOA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12vV7puUjzyuxvoUYvc8dASE6bQwPTFeZSW4ZakTbBYWDfGtQoV, address:104.36.82.9:4016): protocol: rpc: dial tcp 104.36.82.9:4016: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786067275 (26h25m43.567000654s) [noise=true , address=104.36.82.50:4047 , piece_id=4EWCAJZJL6LIUSYU3R7VTUD6D5PTLANB3JMXLXEDNCAVOA5YEBYA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:126Z6tvp49dLLdj3T5snJc4tBVGzcX5UBaGafFN3PkhF92vxpfs, address:104.36.82.50:4047): protocol: rpc: dial tcp 104.36.82.50:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786097369 (26h25m43.567030756s) [noise=true , address=104.36.82.16:4301 , piece_id=TYJHHPC45N5ABFEBZOKW7BYHIKW76TLGEWP6XMH3T4TO47VS3SBA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG, address:104.36.82.16:4301): protocol: rpc: dial tcp 104.36.82.16:4301: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786130870 (26h25m43.567064249s) [noise=true , address=104.36.82.12:4184 , piece_id=GPDRBM723G7SN4QCFGUN6X4DLWVL6Y3EWH3VSG53X4RE5ILXVUGA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:17JJjwwgSn7A6BS7n1a7fphX8oTEQpXmad25ifM2nDhWhPv4Ls, address:104.36.82.12:4184): protocol: rpc: dial tcp 104.36.82.12:4184: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786177517 (26h25m43.567110893s) [noise=true , address=104.36.82.54:4163 , piece_id=XXYKDREEEU3MIRYTTSZMJHWUI4WZAQ3JLO3UX4A732E42V4XRBQA, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: rpc: dial tcp 104.36.82.54:4163: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786220037 (26h25m43.567153418s) [noise=true , address=104.36.82.52:4120 , piece_id=N2WBQD4TF5MYSCK6OW7DDU45S6VWBRCFR5VHUIRYKWLAGLOBT3QQ, seg_pos={0 0}, upload=25207]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 159.223.143.39:60126->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:27.786277291 (26h25m43.567210675s) [seg_pos={0 0}, upload=25207]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:17:27.817659788 (26h25m43.598593162s) [upload=25207]: Done uploading segments... Jul 28 00:17:27.817713581 (26h25m43.598646958s) [upload=25207]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.036063982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 207.560643ms FAIL (2025-07-28 00:16:55 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 207.6767ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 463.721553ms FAIL (2025-07-28 00:16:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 463.813559ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X...XX.] duration: 639.017655ms FAIL (2025-07-28 00:16:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 639.054544ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [................................................XXX.XXXXXXXX] duration: 1m0.036761385s FAIL (2025-07-28 00:16:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== .36.82.9:4026 , piece_id=ROOWLE5W5DD6SPW62XAWBGP436BFP26RL4LPRR7GPYMI36RDXLWQ, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.925662103 (26h23m43.706595492s) [noise=true , address=104.36.82.12:4162 , piece_id=HOATZMHA236S6HHI3GRXWEIF6SJBGCYJ3VI25MUD3TSX4C4USWDQ, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.925826790 (26h23m43.706760222s) [noise=true , address=79.127.201.215:4350 , piece_id=HKUPEYZUTGTZH7GELY5MJ3S7JJ77VTP2YCPKIKR5TLTBRQYBMD6Q, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.926169705 (26h23m43.707103093s) [noise=true , address=104.36.82.20:4440 , piece_id=EEKZUVMCSDTKUZY3LE7TCRKCZUTQ7NFO4GID7LRPP4C5ONOHHU4A, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.926624569 (26h23m43.707557956s) [noise=true , address=104.36.82.46:4493 , piece_id=YFMQRRRFWCVYUWHDLLUCZHCSRFHUFTGFLJHTF6RMA4WXDLE4NYWQ, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.927148024 (26h23m43.708081412s) [noise=true , address=104.36.82.56:4238 , piece_id=3XDAZIPJ2IVVS2X2BOM72IIUC5MLR4ZXPESZBJYUD23MSDDBKVJA, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:27.929483114 (26h23m43.710416501s) [noise=true , address=104.36.82.42:4822 , piece_id=3ZOW7G3FKFZYENB5TD6E7N7UWOGZ6ZG77RIRBFIHIXZCLU3CK5DA, seg_pos={0 0}, upload=25193]: Uploading piece... Jul 28 00:15:28.151750613 (26h23m43.932684003s) [upload=25193]: Next returned nil segment Jul 28 00:15:28.151774469 (26h23m43.932707857s) [upload=25193]: Waiting for error group managing segments... Jul 28 00:15:31.573441393 (26h23m47.354374763s) [noise=true , address=64.49.13.138:4482 , piece_id=RVVBXQSP4QNQ675TMHFJ2VVJ74PP75OVPQD6HAUDBNGWRH3YR6IA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: <nil> Jul 28 00:15:31.683482357 (26h23m47.464415723s) [noise=true , address=79.127.201.215:4350 , piece_id=HKUPEYZUTGTZH7GELY5MJ3S7JJ77VTP2YCPKIKR5TLTBRQYBMD6Q, seg_pos={0 0}, upload=25193]: Done uploading piece. err: <nil> Jul 28 00:16:27.686476811 (26h24m43.467410201s) [noise=true , address=104.36.82.21:4485 , piece_id=JRNTRN5WRXPVPNABRZJLXV3MKQ45TFCFRNMKDWU5MKMSVVVTB26A, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1wmohWyFaMkNm8KBAZprmZ6RuiH6UcB8KRNght2HGpV3Ucnnq9, address:104.36.82.21:4485): protocol: write tcp 159.223.143.39:56582->104.36.82.21:4485: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686567604 (26h24m43.467500984s) [noise=true , address=104.36.82.15:4288 , piece_id=Z2XYI2UO5O6N6G7UFN2GNHQHJTKVSLRJC4BMABL53VMFB32URB7Q, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1gPjPGuueZnizLT2QvJN9y7riwKiW5TWMkQtmGGPP7VZvkNoB7, address:104.36.82.15:4288): protocol: write tcp 159.223.143.39:60850->104.36.82.15:4288: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686637887 (26h24m43.467571251s) [noise=true , address=104.36.82.9:4026 , piece_id=ROOWLE5W5DD6SPW62XAWBGP436BFP26RL4LPRR7GPYMI36RDXLWQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1M7HJBLoxHkRHXbPuDiqwxdb4EAq5wktfJY5FygdoDdgy59rRq, address:104.36.82.9:4026): protocol: write tcp 159.223.143.39:38716->104.36.82.9:4026: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686649054 (26h24m43.467582438s) [noise=true , address=104.36.82.16:4301 , piece_id=MAE3ULPMAGTXUEUJTN3HRU5MIXCCXXHLAPUQDGQ6LFDLMWHIGBOA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG, address:104.36.82.16:4301): protocol: rpc: dial tcp 104.36.82.16:4301: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686706874 (26h24m43.467640251s) [noise=true , address=104.36.82.46:4493 , piece_id=YFMQRRRFWCVYUWHDLLUCZHCSRFHUFTGFLJHTF6RMA4WXDLE4NYWQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 159.223.143.39:48368->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686775467 (26h24m43.467708898s) [noise=true , address=104.36.82.56:4238 , piece_id=3XDAZIPJ2IVVS2X2BOM72IIUC5MLR4ZXPESZBJYUD23MSDDBKVJA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:12U2euZtUdrPZGo2PieqDnKtSftgtfdTRk6V8VfYS4yUW5z3qM5, address:104.36.82.56:4238): protocol: write tcp 159.223.143.39:41062->104.36.82.56:4238: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686836459 (26h24m43.467769835s) [noise=true , address=104.36.82.52:4095 , piece_id=ZAK6REHMKJRGFSQZN7JYFE2VELV56YHBVO7447PGZ5YKDP4M4JLQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1idvNc8SGRP7tkAckr2TLv7LnvftiRUTGcwy5GCrH5rVyDEK5N, address:104.36.82.52:4095): protocol: write tcp 159.223.143.39:51398->104.36.82.52:4095: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686897167 (26h24m43.467830546s) [noise=true , address=104.36.82.10:4085 , piece_id=AUKDTOBTBCZV536ATUWWBMUYT3R4Z3ISBHQYCMJLKTHRUONWFYGQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:18ifvnz4vwaopBJQbF88B1N6DGFmEtfH4jhD3YcneLXCPj7KVu, address:104.36.82.10:4085): protocol: write tcp 159.223.143.39:33584->104.36.82.10:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686969718 (26h24m43.467903096s) [noise=true , address=104.36.82.11:4139 , piece_id=BC3G5RZFYEUZZSTVIDTVIT4LPEJSRY6LCASLDEEP547IMST6A5NQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:4139): protocol: write tcp 159.223.143.39:59266->104.36.82.11:4139: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.686981934 (26h24m43.467915298s) [noise=true , address=104.36.82.13:4210 , piece_id=E37AKJU5GOPR7LYQNSRFLRVNKVQ6WF7P4T3LLVPL6C6DWFLS6WGA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1ybrcx8SPgyEsJWq2P1KYaTwTj5v5BfENKNEZZxaKRvKvuqTiL, address:104.36.82.13:4210): protocol: write tcp 159.223.143.39:46158->104.36.82.13:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687001338 (26h24m43.467934714s) [noise=true , address=104.36.82.12:4162 , piece_id=HOATZMHA236S6HHI3GRXWEIF6SJBGCYJ3VI25MUD3TSX4C4USWDQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1yxXQkfWVGwBuptpPjFgjhfTraAmE3WqtS3WEDbhQsJftjeaxV, address:104.36.82.12:4162): protocol: write tcp 159.223.143.39:33124->104.36.82.12:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687099652 (26h24m43.468033029s) [noise=true , address=104.36.82.13:4211 , piece_id=ZGHGK24JBYSY3LULAROUGMBKGJQ6UGJKT6LIXLDCJLZB5GOWD2MA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:12dANoYTygpcpWfCcAJtsnVBFWSekLhDjheAXnsaYMChRYAm2qc, address:104.36.82.13:4211): protocol: write tcp 159.223.143.39:42726->104.36.82.13:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687118317 (26h24m43.468051679s) [noise=true , address=104.36.82.14:4237 , piece_id=OXW7X4GDRZG7GDXFVQWWJGTFH5PSCEIKQCP3KCUU4YO62647NJEQ, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:104.36.82.14:4237): protocol: write tcp 159.223.143.39:52348->104.36.82.14:4237: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687155178 (26h24m43.468088554s) [noise=true , address=104.36.82.42:4822 , piece_id=3ZOW7G3FKFZYENB5TD6E7N7UWOGZ6ZG77RIRBFIHIXZCLU3CK5DA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:127uUPZhKGPaiMVxTEH919M1WvLqDM9VLgaBkuaTBuPbdaXKLLG, address:104.36.82.42:4822): protocol: write tcp 159.223.143.39:46948->104.36.82.42:4822: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687182023 (26h24m43.468115386s) [noise=true , address=104.36.82.5:4849 , piece_id=VDSTIO7PESNO46YJW3UDLUXWKMJKQ4KLKCRGWYRQI42RQQJGWGHA, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1, address:104.36.82.5:4849): protocol: write tcp 159.223.143.39:38158->104.36.82.5:4849: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687215640 (26h24m43.468149002s) [noise=true , address=104.36.82.20:4440 , piece_id=EEKZUVMCSDTKUZY3LE7TCRKCZUTQ7NFO4GID7LRPP4C5ONOHHU4A, seg_pos={0 0}, upload=25193]: Done uploading piece. err: ecclient: upload failed (node:12GfuKp6kJEy1NTHpo9i21UDm62hrAtKq7124Z6FUehNyyLQ33i, address:104.36.82.20:4440): protocol: rpc: dial tcp 104.36.82.20:4440: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:27.687241549 (26h24m43.468174929s) [seg_pos={0 0}, upload=25193]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:16:27.712793754 (26h24m43.493727138s) [upload=25193]: Done uploading segments... Jul 28 00:16:27.712827277 (26h24m43.493760665s) [upload=25193]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.036790795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 312.016888ms FAIL (2025-07-28 00:16:23 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 312.051914ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X...XX] duration: 30.03396293s FAIL (2025-07-28 00:16:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.034013776s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X...X] duration: 30.031163038s FAIL (2025-07-28 00:15:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031189908s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 447.753696ms FAIL (2025-07-28 00:15:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 447.789913ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 420.6864ms FAIL (2025-07-28 00:15:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 420.728911ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.................................................XXX.XXXXXXX] duration: 1m0.032540778s FAIL (2025-07-28 00:15:13 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 0]: Uploading piece... Jul 28 00:14:13.109307738 (26h22m28.89024112s) [noise=true , address=104.36.82.54:4188 , piece_id=DWIXYV5CLBFIXN33MPK23XPUQ5YMRYRGTOHUNY4E3EU4Y4AAHXLQ, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.110312438 (26h22m28.891245819s) [noise=true , address=104.36.82.56:4234 , piece_id=EASMBX2B2SWRQPC53IZURYNDNVED2UKHLOZOONICILEU3C7PC3VA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.111011852 (26h22m28.891945236s) [noise=true , address=104.36.82.17:4341 , piece_id=AIARVMMJRPT5ZYP5CUENWCLC2RKJDC4NW3H4EBUZRWBDWTT7KWLQ, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.112135088 (26h22m28.893177366s) [noise=true , address=104.36.82.15:4270 , piece_id=DWT5GQ6BZP2HRMZQCPMJBPCBQM5EX6V44XTBX3NMVSL3MRT2PMUA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.113076416 (26h22m28.894009803s) [noise=true , address=104.36.82.12:4178 , piece_id=RCU6AQEOC5MJQ7AK7PUOYLIHJSVNHVMEDFEBYS3MFR7SFZFNDPMA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.113770830 (26h22m28.894704208s) [noise=true , address=104.36.82.20:4461 , piece_id=WPTUY4RDBU5ZLUW2CCS26TXHXZXZDXOLBMXPGZ7D6EGFL5NJEG3Q, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.114244130 (26h22m28.895177515s) [noise=true , address=104.36.82.10:4089 , piece_id=V2JHR5RUGODRWVGRC355TBTCJQR53NNN3JQK26MD43FBETDEXXDA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.114704162 (26h22m28.895637556s) [noise=true , address=104.36.82.41:4762 , piece_id=LHZOHXQCKRYTDPSSZKMCXPILYF5RBNZ634IM7XQKPH7RQONHULAA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.115162808 (26h22m28.896096189s) [noise=true , address=79.127.201.215:4347 , piece_id=4HXNLJCWQHEFE44TSKE47MPEKXZRHXF3OVLFJEEOPLWEQRTGQQJA, seg_pos={0 0}, upload=25170]: Uploading piece... Jul 28 00:14:13.327044919 (26h22m29.107978306s) [upload=25170]: Next returned nil segment Jul 28 00:14:13.327067921 (26h22m29.108001307s) [upload=25170]: Waiting for error group managing segments... Jul 28 00:14:14.316724863 (26h22m30.097658243s) [noise=true , address=64.49.13.140:4508 , piece_id=5ZLAT7WAUDC54I7L7MHEEMFTSJBPTFDF7VQ726USBSYPUZXKTMIQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: <nil> Jul 28 00:14:14.687386935 (26h22m30.468320304s) [noise=true , address=79.127.201.215:4347 , piece_id=4HXNLJCWQHEFE44TSKE47MPEKXZRHXF3OVLFJEEOPLWEQRTGQQJA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: <nil> Jul 28 00:15:13.044434976 (26h23m28.825368379s) [noise=true , address=104.36.82.52:4110 , piece_id=SYEZSKJ6JKHXPV5OKVQGMLPA2LJ5MVBOCQRPEKUQNNHJ5R4H3OSQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044525852 (26h23m28.825459227s) [noise=true , address=104.36.82.17:4341 , piece_id=AIARVMMJRPT5ZYP5CUENWCLC2RKJDC4NW3H4EBUZRWBDWTT7KWLQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1BfZXtm2GymdBAaEBFaX9jeYbK3PveonNNskUq9cLmuW7vbuu5, address:104.36.82.17:4341): protocol: rpc: dial tcp 104.36.82.17:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044602133 (26h23m28.825535522s) [noise=true , address=104.36.82.41:4762 , piece_id=LHZOHXQCKRYTDPSSZKMCXPILYF5RBNZ634IM7XQKPH7RQONHULAA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b, address:104.36.82.41:4762): protocol: rpc: dial tcp 104.36.82.41:4762: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044677731 (26h23m28.825611112s) [noise=true , address=104.36.82.15:4270 , piece_id=DWT5GQ6BZP2HRMZQCPMJBPCBQM5EX6V44XTBX3NMVSL3MRT2PMUA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB, address:104.36.82.15:4270): protocol: rpc: dial tcp 104.36.82.15:4270: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044753011 (26h23m28.825686404s) [noise=true , address=104.36.82.12:4178 , piece_id=RCU6AQEOC5MJQ7AK7PUOYLIHJSVNHVMEDFEBYS3MFR7SFZFNDPMA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12LjJ3SmaNTQi8U1rPSGmR461UYC2CfMtoZ22SSV97hDuhMwC4V, address:104.36.82.12:4178): protocol: rpc: dial tcp 104.36.82.12:4178: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044823126 (26h23m28.825756514s) [noise=true , address=104.36.82.20:4461 , piece_id=WPTUY4RDBU5ZLUW2CCS26TXHXZXZDXOLBMXPGZ7D6EGFL5NJEG3Q, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12nhwT2THUu6ptumqU9Bz1Nd34FHLewx9vWjGFC34j3Lz3RWfkw, address:104.36.82.20:4461): protocol: rpc: dial tcp 104.36.82.20:4461: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044888270 (26h23m28.825821661s) [noise=true , address=104.36.82.10:4089 , piece_id=V2JHR5RUGODRWVGRC355TBTCJQR53NNN3JQK26MD43FBETDEXXDA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: rpc: dial tcp 104.36.82.10:4089: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.044967360 (26h23m28.825900741s) [noise=true , address=104.36.82.42:4803 , piece_id=D56DL3VWAML63HYCC4Y5YA3CQ5HGFSJU7GHIDDQ33JQXXSADQBBQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:104.36.82.42:4803): protocol: rpc: dial tcp 104.36.82.42:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045036746 (26h23m28.825970122s) [noise=true , address=104.36.82.46:4493 , piece_id=QFDLCNQMCFPPHCHRB2QEAZ3LWPY6TGTP33DIVHQNEWHMQMHJ6MJA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: rpc: dial tcp 104.36.82.46:4493: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045106372 (26h23m28.82603975s) [noise=true , address=104.36.82.6:4888 , piece_id=YUC3BCP4NAAHD52KJR77RZM2FVBC4DWS23TBM2Y3IJTHSHSL4CMQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12HwYszCTcXEHqSNnWJpT11NpKg8cFUvdAqU2wUigsLbxmDFkg6, address:104.36.82.6:4888): protocol: rpc: dial tcp 104.36.82.6:4888: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045176922 (26h23m28.826110327s) [noise=true , address=104.36.82.3:4776 , piece_id=C4QWIJRV5F3EUSWUIMOMRV772IZN5H7NX4452EPE3HYNBPIIHPZQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:104.36.82.3:4776): protocol: rpc: dial tcp 104.36.82.3:4776: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045242772 (26h23m28.82617615s) [noise=true , address=104.36.82.56:4234 , piece_id=EASMBX2B2SWRQPC53IZURYNDNVED2UKHLOZOONICILEU3C7PC3VA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:104.36.82.56:4234): protocol: rpc: dial tcp 104.36.82.56:4234: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045309795 (26h23m28.826243195s) [noise=true , address=104.36.82.53:4134 , piece_id=HYU23YI272VYCRXCIV7KAARYWZOWVI275R2EUKC46KMCYH4R3XIQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: rpc: dial tcp 104.36.82.53:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045375571 (26h23m28.826308951s) [noise=true , address=104.36.82.51:4084 , piece_id=ZASL34H2IXTCIRAE6RT2NNTMIYHHEJSC46WPNW4RW4JXFE2UQ5GA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:121BaY9pfRq2a2sZTqmPA3NrXNh9pnckJj21N13sV4jhkuo2gP9, address:104.36.82.51:4084): protocol: rpc: dial tcp 104.36.82.51:4084: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045441696 (26h23m28.826375086s) [noise=true , address=104.36.82.44:4879 , piece_id=6RBYYWEU35DHRUT2GMCQRR5KI2ADIUAIIKZEQJ7W7P3IMTPPUMJA, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:12QCXnCqsnF4z6YenY9FPGC6BSN5EXuQ6VVA9F42KqEx2Bgxf8b, address:104.36.82.44:4879): protocol: rpc: dial tcp 104.36.82.44:4879: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045506193 (26h23m28.826439579s) [noise=true , address=104.36.82.54:4188 , piece_id=DWIXYV5CLBFIXN33MPK23XPUQ5YMRYRGTOHUNY4E3EU4Y4AAHXLQ, seg_pos={0 0}, upload=25170]: Done uploading piece. err: ecclient: upload failed (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG, address:104.36.82.54:4188): protocol: write tcp 159.223.143.39:50888->104.36.82.54:4188: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:13.045528582 (26h23m28.82646199s) [seg_pos={0 0}, upload=25170]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:15:13.072639623 (26h23m28.853573015s) [upload=25170]: Done uploading segments... Jul 28 00:15:13.072691182 (26h23m28.853624558s) [upload=25170]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.032564381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 234.34641ms FAIL (2025-07-28 00:15:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 234.376342ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 378.305215ms FAIL (2025-07-28 00:14:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 378.335918ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 388.290616ms FAIL (2025-07-28 00:14:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 388.357713ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 437.450553ms FAIL (2025-07-28 00:14:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== rue , address=104.36.82.6:4873 , piece_id=KLIOIPDMSH43MPPYETZKI76CHTZZYKHSLXQHZ4BUMTAIHB2UDUFA, seg_pos={0 0}, upload=25145]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3): protocol: rpc: dial tcp 104.36.82.6:4873: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:19.224827526 (26h20m35.005760889s) [seg_pos={0 0}, upload=25145]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:12:19.224911362 (26h20m35.005844727s) [upload=25145]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:12:19.224918336 (26h20m35.005851717s) [upload=25145]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:12:19.362327161 (26h20m35.143260528s) [upload=25145]: Done uploading segments... Jul 28 00:12:19.362346149 (26h20m35.143279514s) [upload=25145]: Upload finished. err: <nil> Jul 28 00:12:51.446334860 (26h21m7.227268227s) [upload=25149]: Starting upload Jul 28 00:12:51.446464358 (26h21m7.22739772s) [upload=25149]: Done starting upload Jul 28 00:12:51.446704322 (26h21m7.227637707s) [upload=25149]: Uploading segments... Jul 28 00:12:51.446787742 (26h21m7.227721116s) [upload=25149]: Got next segment. Inline: false Jul 28 00:12:51.446833329 (26h21m7.227766709s) [upload=25149]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:12:51.446840824 (26h21m7.227774219s) [upload=25149]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:12:51.485529866 (26h21m7.26646325s) [seg_pos={0 0}, upload=25149]: Begin upload segment... Jul 28 00:12:51.485678448 (26h21m7.266611834s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486282709 (26h21m7.267216075s) [seg_pos={0 0}, upload=25149]: Done begin upload segment. Jul 28 00:12:51.486329800 (26h21m7.267263182s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486415444 (26h21m7.267348831s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.486711407 (26h21m7.267644781s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487045013 (26h21m7.267978389s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487052683 (26h21m7.267986065s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487333539 (26h21m7.268266915s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487624449 (26h21m7.268557836s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487637388 (26h21m7.268570764s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.487963294 (26h21m7.268896672s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488384188 (26h21m7.269317573s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488402235 (26h21m7.269335622s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488790390 (26h21m7.269723765s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.488981948 (26h21m7.269915335s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489139202 (26h21m7.270072587s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.489529535 (26h21m7.270462922s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Uploading piece... Jul 28 00:12:51.590540290 (26h21m7.371473675s) [upload=25149]: Next returned nil segment Jul 28 00:12:51.590561833 (26h21m7.371495207s) [upload=25149]: Waiting for error group managing segments... Jul 28 00:12:51.593971573 (26h21m7.374904937s) [noise=true , address=45.91.116.12:4226 , piece_id=R43ZVZ7HUPUFPGSPYRCSZMN3LB4WSKIIUNGNZJQKNDWW27GK7APQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594092106 (26h21m7.375025476s) [noise=true , address=45.91.116.4:4937 , piece_id=TDDSEA6WJHAYPD4RITP2W5BOJS55L2DQOSTRBJHLIDNMAAX4L5SA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.594518922 (26h21m7.375452292s) [noise=true , address=45.91.116.5:4298 , piece_id=A7V4OE6KOWXV63QSDCGNLXPE32NIL7MLGJMZTLGRZHIG3FJQLCYA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596497721 (26h21m7.377431102s) [noise=true , address=45.91.116.9:4263 , piece_id=55Q5ELBLWNPMVSJ2GYRJ5NCOZF75JWAOMOPX7OQRHYNFWG4SU5JQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.596767382 (26h21m7.377700774s) [noise=true , address=77.90.31.15:4699 , piece_id=V6MLSEIOGLEQNKZCOTF7Y3CCBD5ATG7JHFSKTYCC53E4SJZO5GUQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597122347 (26h21m7.378055738s) [noise=true , address=77.90.31.18:4805 , piece_id=YACHLGLUW37BOIBF36YS47THEVYNYWY7TTRN3VTWM5SKNMC5QIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597394157 (26h21m7.378327548s) [noise=true , address=45.91.116.5:4297 , piece_id=EQSG7A6M5GVB7SSAP6P5RHLITZKDFGQCKDB6JSJJ5XV7II25BUVA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597631346 (26h21m7.378564711s) [noise=true , address=77.90.31.10:4519 , piece_id=VR3BU5O2ZGM4OHJTLWU4AWGSSJA26THN5KXXAI5RZR2WOAGGFIFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.597978444 (26h21m7.378911823s) [noise=true , address=77.90.31.11:4555 , piece_id=5ODJAJ4HZBJYYMQ2U6GPZYC7TNCJ7P3EQDIWAHA6AY2I4AIXHPUA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598114353 (26h21m7.379047716s) [noise=true , address=45.91.116.7:4011 , piece_id=AONDOO2JZKGYJI6MB7BVMBXUPFQOMYZYDGBAS23VMSRSNJPYKKSA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598550692 (26h21m7.379484054s) [noise=true , address=45.91.116.10:4083 , piece_id=H3MU35YXKQ5HXXXYQ7L3EQXMSKW2BOLDS5JFR4DRD6QFQEDRHNMA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:12:51.598693441 (26h21m7.37962682s) [noise=true , address=45.91.116.11:4117 , piece_id=CZ6EZULUJSMFUNCDCE76T4TQVI7UXLTU7MGBE3OKQB3GKNWBUBDA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: <nil> Jul 28 00:13:21.415422049 (26h21m37.196355417s) [noise=true , address=104.36.82.2:4730 , piece_id=LPXMNPGLJ6D4VSUKQ6UQQLPKLWR4ADAJZHPR37T3LUVLCDYKWZMQ, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: write tcp 159.223.143.39:39116->104.36.82.2:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415498914 (26h21m37.19643229s) [noise=true , address=104.36.82.16:4294 , piece_id=ZWXMPUMPS3YZFP4YCTOT3OPY45PNRBLKJ6IFABLFANJ4HJ5W5DFA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415564035 (26h21m37.196497411s) [noise=true , address=104.36.82.15:4259 , piece_id=7WJ4YW33BSSBNMMSPNOZN3IUUTVBRKTDHXG4TEK6R2QFIJNWONQA, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:21.415610198 (26h21m37.196543564s) [noise=true , address=104.36.82.4:4804 , piece_id=RDQSGNNW3AU6UIWGK2OHJSLKMZPDELUOU2T2ANDO26F4ND6B6A4Q, seg_pos={0 0}, upload=25149]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:58476->104.36.82.4:4804: use of closed network connection; conn closed: op Jul 28 00:13:21.415628115 (26h21m37.19656148s) [seg_pos={0 0}, upload=25149]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:13:21.442171782 (26h21m37.22310515s) [upload=25149]: Done uploading segments... Jul 28 00:13:21.442216456 (26h21m37.223149832s) [upload=25149]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 437.481769ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 227.924102ms FAIL (2025-07-28 00:14:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 227.955251ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..................................................XXX.XXXXXX] duration: 1m0.036397337s FAIL (2025-07-28 00:14:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== .36.82.12:4174 , piece_id=MUBJTG2N5Q7CITTYLK6PBKK34QE7PKSRLCE544T23FNV6A4LJEPQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.981704249 (26h21m28.76263763s) [noise=true , address=104.36.82.6:4901 , piece_id=OI5GF6QFN7WOH52NWYJCZECLY5QDYFD5JGXOGRJMEXCEQVGQURBQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.982024129 (26h21m28.762957507s) [noise=true , address=104.36.82.15:4279 , piece_id=PVYGX4X3QUGD3D6TD2NXENMLKHRQO5QTKXEAPN4CH4OWV3EXEFYQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.982367533 (26h21m28.763300914s) [noise=true , address=104.36.82.16:4316 , piece_id=RXGU7UI3BQE4VL2TQAYEUZUB6QCJ2JG6ALIIJAFKJFI74EAWKEHA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.982715686 (26h21m28.763649065s) [noise=true , address=104.36.82.14:4241 , piece_id=DLCOMMAKIULBHDCXOT75YUO5PAOJFRZW4UCGPHWQ5DW2CXEB2LLA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.982997054 (26h21m28.763930433s) [noise=true , address=104.36.82.53:4156 , piece_id=TYYK7PPV2HXY3XIIEYB355AR5TS7ETLUWDUU3JNS66G32NPR64BQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.983369414 (26h21m28.764302805s) [noise=true , address=104.36.82.54:4165 , piece_id=FBXBURTZVPMXJE63UPPDDP6YPBDKOJCCSDWDR5U7AB4WVWUJDHJA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.983732452 (26h21m28.76466583s) [noise=true , address=104.36.82.42:4814 , piece_id=GYTSXV42APDBOAUGUBHLSYO5H57KAOLXE3QL7UXH4LCNDSGITSUA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.984085942 (26h21m28.76501932s) [noise=true , address=79.127.201.214:4291 , piece_id=WSWGLWB3QCDN7LVAYQ4PMHNXAIRNQUN2EC27MMH72AXF3ZOTLYUQ, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:12.984485560 (26h21m28.765418945s) [noise=true , address=104.36.82.56:4251 , piece_id=YA4VCKOQA5E2B7TF25I4JQWNRNKXSNC6E23W3XY77LPTF23WG2JA, seg_pos={0 0}, upload=25157]: Uploading piece... Jul 28 00:13:13.189966174 (26h21m28.970899552s) [upload=25157]: Next returned nil segment Jul 28 00:13:13.189983326 (26h21m28.970916692s) [upload=25157]: Waiting for error group managing segments... Jul 28 00:13:13.621200687 (26h21m29.402134056s) [noise=true , address=79.127.201.214:4291 , piece_id=WSWGLWB3QCDN7LVAYQ4PMHNXAIRNQUN2EC27MMH72AXF3ZOTLYUQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: <nil> Jul 28 00:13:14.450347470 (26h21m30.231280849s) [noise=true , address=64.49.13.138:4483 , piece_id=7HHKAGCEXD4KUVEOTIQW4YK6MRR57LQJMXYRZHS6TJO75CCIMH7Q, seg_pos={0 0}, upload=25157]: Done uploading piece. err: <nil> Jul 28 00:13:14.538549427 (26h21m30.319482811s) [noise=true , address=79.127.201.215:4343 , piece_id=2FPWHHI4O366U6WRP2PRVUGEKEL2F7A2WQAYZXYPVOYPA7PNGN3Q, seg_pos={0 0}, upload=25157]: Done uploading piece. err: <nil> Jul 28 00:14:12.912102155 (26h22m28.693035531s) [noise=true , address=104.36.82.16:4316 , piece_id=RXGU7UI3BQE4VL2TQAYEUZUB6QCJ2JG6ALIIJAFKJFI74EAWKEHA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12PPnXDzwXDVDKemw483GXuzQrPN48JJrjeeD7az2NPNawHNaXf, address:104.36.82.16:4316): protocol: rpc: dial tcp 104.36.82.16:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912253520 (26h22m28.693186895s) [noise=true , address=104.36.82.20:4447 , piece_id=CXGMHB62TKJ6NIYMVMGIIINYXHGMTD23BBHBYZUHGCM46ZZ2VUFQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:1Li8ZwrWGPx5Y478pYLwnWJ7N2x3dbXgwGaUGWVtVhvv61EnTs, address:104.36.82.20:4447): protocol: rpc: dial tcp 104.36.82.20:4447: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912347582 (26h22m28.693280968s) [noise=true , address=104.36.82.4:4825 , piece_id=6PCCIUN63YUI4VXDRIVM5C5FQXB7B4NBYJ57QNRDUSP6NCWNKFDA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:121Bec7R14ujGEdoUqFTikngLVLb6jmLAcHkeBAaCkhZ5dr6APb, address:104.36.82.4:4825): protocol: rpc: dial tcp 104.36.82.4:4825: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912449723 (26h22m28.693383098s) [noise=true , address=104.36.82.5:4872 , piece_id=YM6BND267SZEK2HOE7AHCU7LIGJLJ2P7PLV4EVCPACCDJZQZKFCA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:1zqxiFpqKeXT9mhXUDuZZmvJkhejC8gcHTQgTadY1xUpQyLyZi, address:104.36.82.5:4872): protocol: write tcp 159.223.143.39:50330->104.36.82.5:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912653063 (26h22m28.693586453s) [noise=true , address=104.36.82.42:4814 , piece_id=GYTSXV42APDBOAUGUBHLSYO5H57KAOLXE3QL7UXH4LCNDSGITSUA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12rnmX5iuA5swKceTZVV14HLaUctRFimn3DPLjRogg4K3N7PWnU, address:104.36.82.42:4814): protocol: rpc: dial tcp 104.36.82.42:4814: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912722923 (26h22m28.693656311s) [noise=true , address=104.36.82.56:4251 , piece_id=YA4VCKOQA5E2B7TF25I4JQWNRNKXSNC6E23W3XY77LPTF23WG2JA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12vvYKd1KXrz8cYXRJSL9KXiYQtY95aB2GFPGFi9c9Nj1Raj6J4, address:104.36.82.56:4251): protocol: rpc: dial tcp 104.36.82.56:4251: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912786120 (26h22m28.693719481s) [noise=true , address=104.36.82.15:4279 , piece_id=PVYGX4X3QUGD3D6TD2NXENMLKHRQO5QTKXEAPN4CH4OWV3EXEFYQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: rpc: dial tcp 104.36.82.15:4279: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912854267 (26h22m28.693787634s) [noise=true , address=104.36.82.14:4241 , piece_id=DLCOMMAKIULBHDCXOT75YUO5PAOJFRZW4UCGPHWQ5DW2CXEB2LLA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12aBy6nQScbjzmBSScfEZwTX8NCcP9c71FRQkUH54WvZhhhtXLR, address:104.36.82.14:4241): protocol: rpc: dial tcp 104.36.82.14:4241: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912911508 (26h22m28.693844877s) [noise=true , address=104.36.82.13:4202 , piece_id=ZNMT57OYPDKVYPTMXLDUCEPFGF24ZPDTPM64GDABVXZM76ZHFISA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:18wf5KnaR6WaR7MJivJkMJ7FejbuR6KnyvtRQgazNKfDb8fbsH, address:104.36.82.13:4202): protocol: rpc: dial tcp 104.36.82.13:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.912954899 (26h22m28.693888274s) [noise=true , address=104.36.82.54:4165 , piece_id=FBXBURTZVPMXJE63UPPDDP6YPBDKOJCCSDWDR5U7AB4WVWUJDHJA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913027995 (26h22m28.693961371s) [noise=true , address=104.36.82.52:4100 , piece_id=6YR6JR2G3QGHMWFXD36HVW6VA5ZM2T7AOPV3BYLFK2VMR4G4KTDQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12JQ78Lg28PborUyvkyLJb5RikJ8zJSAT5GkpYSsZLc6HRmkhyw, address:104.36.82.52:4100): protocol: rpc: dial tcp 104.36.82.52:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913037598 (26h22m28.693970974s) [noise=true , address=104.36.82.6:4901 , piece_id=OI5GF6QFN7WOH52NWYJCZECLY5QDYFD5JGXOGRJMEXCEQVGQURBQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: rpc: dial tcp 104.36.82.6:4901: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913113686 (26h22m28.694047063s) [noise=true , address=104.36.82.55:4226 , piece_id=WVMP3URSBIFGUTHGYVAZHIMITT6PB6TO6IJJ36QNBRC65ISWAXYA, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:1qN8jYqiZGuPjqHSp6siqdT4p33qvf1vYdAkMZKRnFgcQj9Ga3, address:104.36.82.55:4226): protocol: write tcp 159.223.143.39:35470->104.36.82.55:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913119763 (26h22m28.694053131s) [noise=true , address=104.36.82.12:4174 , piece_id=MUBJTG2N5Q7CITTYLK6PBKK34QE7PKSRLCE544T23FNV6A4LJEPQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12ZqPqvh4SVpDmkpVKQXZUW48CtzV9VMeMGJoDZwA7KpNfmjfow, address:104.36.82.12:4174): protocol: write tcp 159.223.143.39:32996->104.36.82.12:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913177711 (26h22m28.694111087s) [noise=true , address=104.36.82.53:4156 , piece_id=TYYK7PPV2HXY3XIIEYB355AR5TS7ETLUWDUU3JNS66G32NPR64BQ, seg_pos={0 0}, upload=25157]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 159.223.143.39:60108->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:12.913210562 (26h22m28.694143938s) [seg_pos={0 0}, upload=25157]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:14:12.945831453 (26h22m28.72676483s) [upload=25157]: Done uploading segments... Jul 28 00:14:12.945859681 (26h22m28.726793062s) [upload=25157]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.036431268s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)