Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...........................X.] duration: 687.312986ms trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:32:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 687.333998ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...............................X.] duration: 373.159223ms trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:32:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 373.187205ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...........................X] duration: 1m0.027525358s trace: https://jaeger.dev.storj.io/trace/5e959267a485eb67 FAIL (2025-11-28 22:32:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 7.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260643895 (420h44m42.02832733s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.260661524 (420h44m42.028344955s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261110284 (420h44m42.02879371s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261157035 (420h44m42.028840468s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261503954 (420h44m42.029187382s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261621464 (420h44m42.029304897s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.261902773 (420h44m42.0295862s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262150516 (420h44m42.029833946s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262344635 (420h44m42.030028068s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262575897 (420h44m42.030259337s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262686666 (420h44m42.030370094s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.262971189 (420h44m42.030654659s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263047035 (420h44m42.030730461s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263270751 (420h44m42.030954179s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.263560200 (420h44m42.031243636s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.264950890 (420h44m42.032634315s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.265080971 (420h44m42.032764382s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Uploading piece... Nov 28 22:31:17.329096045 (420h44m42.096779464s) [upload=437903]: Next returned nil segment Nov 28 22:31:17.329108757 (420h44m42.096792188s) [upload=437903]: Waiting for error group managing segments... Nov 28 22:31:17.334237280 (420h44m42.101920695s) [noise=true , address=145.79.227.9:4376 , piece_id=BER2DFV3QPCXRO3QUVRJRVT6G7WWRW3NDN4LVWEI5P3EPFX7JS3Q, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.335587338 (420h44m42.103270764s) [noise=true , address=145.79.227.46:4617 , piece_id=V4ESXDCC5HAJMNVZDDPQWURC6TP7F6FO3QO4U3KB5JXR3CRTILCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336047163 (420h44m42.103730588s) [noise=true , address=145.79.227.20:4467 , piece_id=T4LDHMLX5YOUT3TPYKHZWL3PHV73CT742GHM5HAIUCFTZVQQHIVQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336411376 (420h44m42.104094802s) [noise=true , address=145.79.227.3:4183 , piece_id=XV57THCFKPOEPMJMAQ7QT4GCQYK7MMKAW4EUFXBJ7MDSA7THGQUA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.336847189 (420h44m42.104530614s) [noise=true , address=145.79.227.33:4872 , piece_id=TSU7OQDCFGCL64WT7NP2UZN4JNJXBPAS7RXHQYWFVIRIJ27RCGIA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337169495 (420h44m42.104852924s) [noise=true , address=145.79.227.4:4194 , piece_id=ATDRL56IEWUBKG5T5NRJGRI4WAWUFBN4ADFZYAX7A6N5F3QDD5RA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.337701888 (420h44m42.105385304s) [noise=true , address=145.79.227.12:4800 , piece_id=64PE4E56ZZ7PQE2KQ3LQDHJBJPORIOS6QUM3FO5TVCDR5SORYHGQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.346841936 (420h44m42.114525361s) [noise=true , address=145.79.227.11:4764 , piece_id=R74XNCB7TT4B6SUWBAI3DUTSAJ3ZOHTD23RZSEDYPE7UGWYIGHOA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.615483048 (420h44m42.383166482s) [noise=true , address=64.49.13.169:4715 , piece_id=NQEWXOVXB7XULFMTG2VRMSIWPK5AAK3XLSEZZI7PNXTDTYTODM7A, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1zHWknNPWJ79pDmU1vfv8TCqWKU9sxrP4upTRBNsPnc8LcLgzK, address:64.49.13.169:4715): protocol: storage node overloaded, request limit: 2000 Nov 28 22:31:17.946111222 (420h44m42.713794648s) [noise=true , address=64.49.13.137:4494 , piece_id=SSRO4MW2JLF5TCEFGZYDABRINDYQUKTLJ4G577EQJU7RJZZ2LHCA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:31:17.952915451 (420h44m42.720598875s) [noise=true , address=64.49.13.170:4728 , piece_id=OJCJS5YBZTSUJ7SUSCG3BUMUI47SK2POTTBTUSL7U5NZ3S5ZO7PQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: <nil> Nov 28 22:32:17.196604220 (420h45m41.96428772s) [noise=true , address=77.90.31.21:4911 , piece_id=7EHCJBEIJOWW5ECNCUZCQAP3CQEBABOSVBS2N6Z27I4IA5APAGBA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1cxBiYLx8eWayRmXGXzNyRKAMLEcL7gJ7VU4wLvKhN2nCQbqwf, address:77.90.31.21:4911): protocol: rpc: dial tcp 77.90.31.21:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197072174 (420h45m41.96475561s) [noise=true , address=77.90.31.18:4812 , piece_id=OIB4WMDPFMHHTHYPE64WQLUIGGOCXGWFBJAKADJDJ4MVEA7QULDA, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn, address:77.90.31.18:4812): protocol: rpc: dial tcp 77.90.31.18:4812: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197455798 (420h45m41.965139233s) [noise=true , address=45.91.116.11:4132 , piece_id=OB3PHBE7BRSATKR322TKZWS4NPRFLK4VK4U2NZMBGBGUK6EHH7IQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1T2RDXpgwVh2KFGpjESCMQCdpTufuMe9tSgLHCmnVctHepJyEa, address:45.91.116.11:4132): protocol: rpc: dial tcp 45.91.116.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.197891544 (420h45m41.965574979s) [noise=true , address=45.91.116.12:4229 , piece_id=UCEYEPVUUCV3R4LXSNNEFKGJCQEXS3F64LGEVOJ6NHTVR4WLWPIQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:12rfuAG41swAn82EM5ZTAN31NhGp4UKMHFiZiT57NXmSAi3MHe2, address:45.91.116.12:4229): protocol: rpc: dial tcp 45.91.116.12:4229: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198350042 (420h45m41.966033477s) [noise=true , address=77.90.31.17:4790 , piece_id=EMWIAEMDWCT6QKSQFPFJELK5P7H4VHMDMROGTTQZ6N3PAL7MLIMQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.198523947 (420h45m41.96620738s) [noise=true , address=45.91.116.13:4186 , piece_id=BZLMAHGUBDCK2TTSZHYTWYSZIZS5DAZJK5OHAJIZMZTQJ4EBUJAQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1U2z7FPoTM2387UEPKbaP1NFas5APdPkUzuA2D6BwjKxC57hht, address:45.91.116.13:4186): protocol: read tcp 159.223.143.39:44236->45.91.116.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199518463 (420h45m41.9672019s) [noise=true , address=45.91.116.10:4086 , piece_id=2DPEZ3HAFODVMQZIYW4ZJDP2SDDXU5D6YO2GPUMKBK2ATUDLVVDQ, seg_pos={0 0}, upload=437903]: Done uploading piece. err: ecclient: upload failed (node:1X6eg8EWzh7BPYUaLN1wGYPDfkg1RCKZYqDCcJhCtQXApbsG5A, address:45.91.116.10:4086): protocol: read tcp 159.223.143.39:49844->45.91.116.10:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:17.199559360 (420h45m41.967242772s) [seg_pos={0 0}, upload=437903]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:17.222042557 (420h45m41.989725988s) [upload=437903]: Done uploading segments... Nov 28 22:32:17.222122762 (420h45m41.989806197s) [upload=437903]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.027603515s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...............................X] duration: 1m0.03405067s trace: https://jaeger.dev.storj.io/trace/394c14e93ec383ca FAIL (2025-11-28 22:32:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Nov 28 22:31:13.028161832 (420h44m37.795845256s) [upload=437901]: Starting upload Nov 28 22:31:13.028211560 (420h44m37.795894987s) [upload=437901]: Done starting upload Nov 28 22:31:13.028478006 (420h44m37.796161431s) [upload=437901]: Uploading segments... Nov 28 22:31:13.028687548 (420h44m37.796370981s) [upload=437901]: Got next segment. Inline: false Nov 28 22:31:13.028732172 (420h44m37.796415597s) [upload=437901]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:13.028737642 (420h44m37.796421068s) [upload=437901]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:13.055550216 (420h44m37.823233644s) [seg_pos={0 0}, upload=437901]: Begin upload segment... Nov 28 22:31:13.055616690 (420h44m37.823300114s) [seg_pos={0 0}, upload=437901]: Done begin upload segment. Nov 28 22:31:13.055663363 (420h44m37.823346792s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057487705 (420h44m37.825171137s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.057504070 (420h44m37.825187502s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058015785 (420h44m37.825699213s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058091611 (420h44m37.825775045s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058452544 (420h44m37.826135976s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058509959 (420h44m37.826193393s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058840136 (420h44m37.826523568s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.058895204 (420h44m37.826578639s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059253675 (420h44m37.826937111s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059314042 (420h44m37.826997468s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059676322 (420h44m37.827359753s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.059688812 (420h44m37.827372237s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060052109 (420h44m37.827735536s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060408056 (420h44m37.828091485s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.060794818 (420h44m37.82847825s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061149884 (420h44m37.828833312s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.061184377 (420h44m37.828867808s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Uploading piece... Nov 28 22:31:13.126212343 (420h44m37.893895765s) [upload=437901]: Next returned nil segment Nov 28 22:31:13.126228101 (420h44m37.893911528s) [upload=437901]: Waiting for error group managing segments... Nov 28 22:31:13.131957889 (420h44m37.899641316s) [noise=true , address=145.79.227.6:4284 , piece_id=CKHTK73NEYGQJEKO2PE5JKWZBDPPPVIVDGID5A3SKQJQ72Q4UIRA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132241911 (420h44m37.899925336s) [noise=true , address=145.79.227.3:4163 , piece_id=3HE2M3RDM22B2BU3DQKBIIKOCA6Q2WDO2MM7YY326IAC4WUKQKBQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132517207 (420h44m37.900200633s) [noise=true , address=145.79.227.16:4925 , piece_id=BQIUD5743RQB3W6MMASOKU2JL4IFAVOYEGRPVAUAGKJAKLHGIUIA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.132920536 (420h44m37.900603961s) [noise=true , address=145.79.227.51:4440 , piece_id=YISGZTMALV3I2J4LNNIECJZIF3Q7QLP2MTRC2IQER5FM622BNKDA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.133465618 (420h44m37.901149055s) [noise=true , address=145.79.227.8:4336 , piece_id=L5EYCEZRZRFM4HVF33YQHJGZBEV3YPKTWIJ5FUPMY2BLNLBN7FZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.143585331 (420h44m37.911268756s) [noise=true , address=145.79.227.33:4895 , piece_id=ZKTLNTFZ32IRUM5AFJ2RKMVTQ6FSSORNWIJIXVM7WT5KZWZKDYZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.144396831 (420h44m37.912080256s) [noise=true , address=145.79.227.7:4325 , piece_id=4FDLTOQIEH5EKOTPGPQLNNAAZ35FMMAX6KR3A7XMAZA7OGT3GO6A, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:31:13.147345800 (420h44m37.915029224s) [noise=true , address=145.79.227.33:4879 , piece_id=7PHK3V5Z4H7CUMMD23N32G3CDB2QRNLCUI55KOND73F2I2E7UITQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.020408352 (420h45m26.788091779s) [noise=true , address=77.90.31.11:4569 , piece_id=ENQOS26L7ZS4MCA5VF6UZXWMLGACFTG6RBFQSU5D7ML2NN4YUUSQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:02.498580840 (420h45m27.266264266s) [noise=true , address=45.91.116.11:4144 , piece_id=4JJYZOCRL7CDJGSLHAPF5QHYEBPWH2KNQTI2NQPTP2NDJ3QD6VZQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: <nil> Nov 28 22:32:12.998136375 (420h45m37.765819811s) [noise=true , address=45.91.116.14:4194 , piece_id=T2GQ5VXLBRUJHQ7JVROFWWLKNZGWUGWBVKOTV4V6VWHQGXROX6FA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: rpc: dial tcp 45.91.116.14:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998325040 (420h45m37.76600847s) [noise=true , address=45.91.116.12:4235 , piece_id=6QY7AQ2XKFKIJVSXQKWBREYMNH2HPTLQGQUVAHRSJVGAPKBCHTQA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1zJ9RVdURVEzMruVa7LYGiPmGX4JdTemxSDwYUEbt8bA6U73WB, address:45.91.116.12:4235): protocol: read tcp 159.223.143.39:60354->45.91.116.12:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998398791 (420h45m37.76608223s) [noise=true , address=77.90.31.16:4750 , piece_id=E2MBHY6K6QDHH6KCYLY5AUPLEUB4VDM37E5FPHSZWA2BWHDPJBEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12TGd8NnWdekDtHzBQHmJPgTtHjXC6g8fi6RoZYFQJVHDa5hdyE, address:77.90.31.16:4750): protocol: rpc: dial tcp 77.90.31.16:4750: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998429354 (420h45m37.766112792s) [noise=true , address=77.90.31.19:4856 , piece_id=HNQQPITVELIXHMMOQACMD5T67KDXAUTDSHNKUNY2HM5SFLTCHG3Q, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1MntjJtcCDSRozKpesp6S2ZjYrMhxiQheDiPMPSCnavWBBGYt3, address:77.90.31.19:4856): protocol: write tcp 159.223.143.39:52096->77.90.31.19:4856: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998639206 (420h45m37.766322643s) [noise=true , address=45.91.116.10:4104 , piece_id=LWIBWXL7YH4OQIKDD52VVFQUZ4MJQ4QXMHSJAUSIVUBJI3C53OEA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1Hb2ay2TWn5B9Z6UwGTSkJhpGvodkz293N9T6cxSmeCYsZG3Cj, address:45.91.116.10:4104): protocol: read tcp 159.223.143.39:46094->45.91.116.10:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998736579 (420h45m37.766419995s) [noise=true , address=77.90.31.14:4666 , piece_id=Y2V76P7PM5BE6AWI5OALG5QVP2U3BODRZLZEIWKZREQTODOX4TTQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12E2nbaYSxeiBd8NdLfnpkoZ5ef7M13AfrfpKkyrRtNaE19Whso, address:77.90.31.14:4666): protocol: read tcp 159.223.143.39:39524->77.90.31.14:4666: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.998961966 (420h45m37.766645399s) [noise=true , address=77.90.31.12:4594 , piece_id=VIKK6LHQ54QCYQRMGB6E6HBOTFGORWOPBORCSIBZLZLNR755RIPA, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:1J9Cvfqr5QUrVSoqXVMoVtaXUbvcoXWMFoJXoZyuQreSgQbTtU, address:77.90.31.12:4594): protocol: read tcp 159.223.143.39:45956->77.90.31.12:4594: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999004282 (420h45m37.766687716s) [noise=true , address=77.90.31.21:4906 , piece_id=G3C22NCWFISSS6PANB4AGDOBB52KQZGFF4ALEGHQURCAWKWVNZJQ, seg_pos={0 0}, upload=437901]: Done uploading piece. err: ecclient: upload failed (node:12HG7FfbxD7WJZxWS5CXYFuJgj8n5L7tMr1ZfrcyF6Lih2S67R9, address:77.90.31.21:4906): protocol: rpc: dial tcp 77.90.31.21:4906: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:32:12.999082129 (420h45m37.766765542s) [seg_pos={0 0}, upload=437901]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Nov 28 22:32:13.029104041 (420h45m37.796787469s) [upload=437901]: Done uploading segments... Nov 28 22:32:13.029143342 (420h45m37.79682677s) [upload=437901]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.03407507s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X..............................X] duration: 30.030647801s trace: https://jaeger.dev.storj.io/trace/1338b9a894d626be FAIL (2025-11-28 22:31:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Nov 28 22:31:25.049093367 (420h44m49.816776802s) [upload=437906]: Starting upload Nov 28 22:31:25.049171525 (420h44m49.816854971s) [upload=437906]: Done starting upload Nov 28 22:31:25.049243524 (420h44m49.816926941s) [upload=437906]: Uploading segments... Nov 28 22:31:25.049747039 (420h44m49.817430465s) [upload=437906]: Got next segment. Inline: false Nov 28 22:31:25.049787565 (420h44m49.817470994s) [upload=437906]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:31:25.049792458 (420h44m49.817475887s) [upload=437906]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:31:25.075522918 (420h44m49.843206348s) [seg_pos={0 0}, upload=437906]: Begin upload segment... Nov 28 22:31:25.075582229 (420h44m49.843265655s) [seg_pos={0 0}, upload=437906]: Done begin upload segment. Nov 28 22:31:25.075615594 (420h44m49.843299025s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.075734563 (420h44m49.843417994s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.076754299 (420h44m49.844437736s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077276113 (420h44m49.844959547s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077291835 (420h44m49.84497527s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077731071 (420h44m49.845414507s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.077814282 (420h44m49.845497713s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078252485 (420h44m49.845935922s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078389679 (420h44m49.846073111s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.078915446 (420h44m49.846598876s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079119654 (420h44m49.846803089s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079565461 (420h44m49.847248897s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.079985254 (420h44m49.847668689s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080457688 (420h44m49.848141124s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.080907608 (420h44m49.848591044s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.081683203 (420h44m49.849366616s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Uploading piece... Nov 28 22:31:25.160341701 (420h44m49.928025128s) [upload=437906]: Next returned nil segment Nov 28 22:31:25.160358270 (420h44m49.928041697s) [upload=437906]: Waiting for error group managing segments... Nov 28 22:31:25.168517821 (420h44m49.936201257s) [noise=true , address=145.79.227.43:4159 , piece_id=NB4YJUB4GO4CCBMZX44NVEFTXOXLFUODRYUT7HCEHSI5A52MDDDA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.168741547 (420h44m49.936424976s) [noise=true , address=145.79.227.11:4732 , piece_id=XTZDHPLKWOFZAP6BGMVMYPJQSQ5XGUD5LEAS2VG6LYPXF42FSQJA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169149516 (420h44m49.936832942s) [noise=true , address=145.79.227.18:4006 , piece_id=WAQPAFNM53YKF2TLEDQEBRJSCZ4XQIGAYRSKOELVLC442TTHKNQQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169339599 (420h44m49.937023025s) [noise=true , address=145.79.227.41:4085 , piece_id=TJQVGLGYI77CPYF5WHAQCC5C3DXCZJPQFDOR5BOMYAHFMINU6WUQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169602208 (420h44m49.937285634s) [noise=true , address=145.79.227.15:4876 , piece_id=QOHPQXMECNN6QZXBXNZAVRWQJNFSQCIBZJGPVPAKI7JPF4JAF2HQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.169864403 (420h44m49.937547829s) [noise=true , address=145.79.227.31:4797 , piece_id=MKJUZMLC3W6YZPJVOFCJ3I47GBF3HSLKSFKCVL7A3PEZYCFTVDSQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.171769700 (420h44m49.939453131s) [noise=true , address=145.79.227.39:4013 , piece_id=MID4LBCDC7NL6QOPNCIKU6W72DTYP73KCI3Q5H4J2IC4VTZRXBDQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.172086417 (420h44m49.939769842s) [noise=true , address=145.79.227.22:4510 , piece_id=WD32BHCE245D6U5YQBOKC3C75E5P6X5BWVBVDFYVJMBQQEDQ3MBA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:25.181350053 (420h44m49.949033478s) [noise=true , address=145.79.227.33:4869 , piece_id=CAKMAGFXKAWVMBQG2T5LET7I4JQZAEAFDZO45ZP5UXXBHXQNDP6A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: <nil> Nov 28 22:31:55.016664866 (420h45m19.784348293s) [noise=true , address=45.91.116.4:4939 , piece_id=64F4544WGITG2OHXD6N4MQFWUMJWH3RK5DAU5A2IAEAP24GHX66Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 159.223.143.39:36286->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.016871373 (420h45m19.784554798s) [noise=true , address=77.90.31.11:4555 , piece_id=WM5A3TAMX4MJKDVHD6DO7KE6CNEJNPDAWLSI2XODBRFURXHKMZ7A, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 159.223.143.39:56926->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017166673 (420h45m19.784850101s) [noise=true , address=77.90.31.15:4698 , piece_id=2Z2CVAMVBXF35VAPJKCXCE22VA6S6HGHF4TZYQ2UR5BUDX4HSMTQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 159.223.143.39:60584->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017479715 (420h45m19.785163149s) [noise=true , address=77.90.31.12:4590 , piece_id=OIYUTXK52ZHXXL24GKHRKZ4HVCYJMQU4HZ2RSIGFKIFZFAAGBUAQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017652127 (420h45m19.785335556s) [noise=true , address=45.91.116.12:4227 , piece_id=2ZT5OQS7KBSYEY652ISVUELDB3KAWOLH7OD2LAHXUOMSF5BWN6UA, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: read tcp 159.223.143.39:37982->45.91.116.12:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.017858836 (420h45m19.785542262s) [noise=true , address=77.90.31.16:4735 , piece_id=GJLFZDYIBMKB76CXIJJ6RDMRY662ESTLYIXEOG2JA6RDFKMHQLKQ, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: read tcp 159.223.143.39:37782->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018532551 (420h45m19.786215985s) [noise=true , address=77.90.31.17:4772 , piece_id=6V5DKD52WLGETS5MNYVZKG3J2XUXUIFAPULDZW5CQLGUMUDOE74Q, seg_pos={0 0}, upload=437906]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: rpc: dial tcp 77.90.31.17:4772: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:31:55.018570933 (420h45m19.786254351s) [seg_pos={0 0}, upload=437906]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 28 22:31:55.045462996 (420h45m19.813146421s) [upload=437906]: Done uploading segments... Nov 28 22:31:55.045495202 (420h45m19.813178627s) [upload=437906]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.030667523s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........................XX.............................XX] duration: 5.001428683s trace: https://jaeger.dev.storj.io/trace/30e73aaec70c4394 FAIL (2025-11-28 22:31:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00154592s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................XX.............................X] duration: 5.002468211s trace: https://jaeger.dev.storj.io/trace/4cfccd4aaacad901 FAIL (2025-11-28 22:31:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002541092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 350.212038ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:30:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 350.237363ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 462.412881ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:30:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.451175ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 468.446775ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:30:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 468.491975ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................XX.............................] duration: 426.725035ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:30:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 426.827514ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 555.919699ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:30:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 555.947504ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 391.308417ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:29:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 391.336207ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................XX............................] duration: 435.993363ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:29:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 436.010765ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 507.547341ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:29:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 507.579343ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 586.78934ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:29:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 586.816938ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 502.615628ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:29:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 502.64118ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 410.880963ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:28:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 410.905432ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................XX...........................] duration: 452.029426ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:28:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 452.050768ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 350.420736ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:28:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 350.456227ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 496.736984ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 496.761876ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 377.261579ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:27:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 377.29066ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 557.124905ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:27:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 557.151476ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 445.479769ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:27:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 445.502109ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 860.154058ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:27:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 860.201491ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................XX..........................] duration: 475.042154ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:27:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 475.060155ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 544.883806ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:27:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 544.906688ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................XX.........................] duration: 420.31474ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:26:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 420.343249ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 444.166008ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:26:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 444.193845ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 561.482889ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:26:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 561.511311ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 365.292266ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:26:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 365.317439ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................XX........................] duration: 601.272411ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:26:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 601.318354ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 457.262977ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:25:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 457.409823ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 519.704631ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:25:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 519.732388ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 520.182951ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:25:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 520.204069ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 517.798379ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:25:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 517.841478ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................XX.......................] duration: 446.944309ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:25:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 446.970793ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 358.674492ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:24:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 358.705765ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 530.129082ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:24:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 530.148461ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 357.5247ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:24:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 357.546773ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.02177585s trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:24:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1.02180987s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................XX......................] duration: 448.597536ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:24:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 448.783296ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 442.958542ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:24:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 442.982405ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 357.29655ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:23:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 357.320434ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................XX.....................] duration: 593.569727ms trace: https://jaeger.dev.storj.io/trace/2c31733a68ed7fbe FAIL (2025-11-28 22:23:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 593.592091ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 470.339745ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:23:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 470.362023ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 563.056291ms trace: https://jaeger.dev.storj.io/trace/29b5a7816e4fcd08 FAIL (2025-11-28 22:23:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== upload=437481]: Uploading piece... Nov 28 22:07:16.192657858 (420h20m40.960341283s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193211218 (420h20m40.960894653s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.193756384 (420h20m40.9614398s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194374569 (420h20m40.962057997s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194866866 (420h20m40.962550302s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.194966417 (420h20m40.962649848s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.195708769 (420h20m40.963392214s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196169665 (420h20m40.963853093s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.196338464 (420h20m40.964021881s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Uploading piece... Nov 28 22:07:16.272268541 (420h20m41.039951967s) [upload=437481]: Next returned nil segment Nov 28 22:07:16.272286039 (420h20m41.039969474s) [upload=437481]: Waiting for error group managing segments... Nov 28 22:07:16.277470618 (420h20m41.045154044s) [noise=true , address=145.79.227.15:4897 , piece_id=W3SZWNFYW4SKLWVIECOH5LVFWDDYL75JZH247RPIXHAYSGKDWZSA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.277688612 (420h20m41.045372047s) [noise=true , address=145.79.227.50:4396 , piece_id=BDHU6Y533ECJHW524BG26X7G43F3DDCHJMEMGXWCVJH32LZE7Q6A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278295080 (420h20m41.045978505s) [noise=true , address=145.79.227.19:4088 , piece_id=OJMN3QATQPHEUSXXMKDXOTARWWGJMP4WPUHXTKODJCJNVXNZTTLA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.278801064 (420h20m41.046484502s) [noise=true , address=145.79.227.5:4241 , piece_id=44KEGO33KFDVAXMQKAIG7I6HWNRAUDSFTP7N63NB5R6BLPVZI6TA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.279200870 (420h20m41.046884295s) [noise=true , address=145.79.227.36:4551 , piece_id=3GBSYXYUXTE6ENVE2XJ3AJLFDU6ZCVYZETNGLEHMGNJBKKLP2O4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.291671622 (420h20m41.059355049s) [noise=true , address=145.79.227.46:4635 , piece_id=DKPOL62DS7FPBCZNX3IF4VQ7274FKOADPURRXWCARK4RBNDLBBFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:07:16.824540403 (420h20m41.592223828s) [noise=true , address=64.49.13.170:4728 , piece_id=AWYYFTCX5VUDFXT6UKO2GXCHMPPUS222RNG2ICBM6RVMFSFSBT5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: <nil> Nov 28 22:08:16.106697199 (420h21m40.87438063s) [noise=true , address=77.90.31.10:4540 , piece_id=2Z5JF5JCUP73ZSG2LHZKPXHFHZQITYVYIP2OANH6QLQLWSYUOP4A, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:59208->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106829204 (420h21m40.874512629s) [noise=true , address=45.91.116.12:4243 , piece_id=ROJTOO7AGRURVEAM4SCE7N2GX6VVZ5P5GZKH6KVEWVZIBAWUN6GQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1e9VEMmhYq6PxqvTpxsoUHzqYVwujfj1UG7HSZFeiu8CderMpM, address:45.91.116.12:4243): protocol: read tcp 159.223.143.39:37264->45.91.116.12:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.106895832 (420h21m40.874579263s) [noise=true , address=45.91.116.5:4330 , piece_id=H27H677H5HEEIRR2BGTLYRKAUDMLZ3J5ASKKDN325VPTQMWUXUCA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12MK9aVDRwfWPfckFR6RfPUL1Nrm9bwfsgV7TXCKGbtadHebWvW, address:45.91.116.5:4330): protocol: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:53846->45.91.116.5:4330: use of closed network connection Nov 28 22:08:16.106939942 (420h21m40.874623367s) [noise=true , address=45.91.116.13:4169 , piece_id=CAU6R66NUN3X7PSGI7TTF6YYHFL4NSF4W6MU3SKEKAMX2DB2CTOQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33492->45.91.116.13:4169: use of closed network connection Nov 28 22:08:16.106989771 (420h21m40.874673196s) [noise=true , address=77.90.31.14:4696 , piece_id=XHYTJ6FIMF64JV6FIPS47P4CJEPZHNELSPWJ4MZCODQHJKBWYPFQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12mKMnBWeVnjs8QnyFeHTMX8ekvKN2UbK2W2L9KqAY4V136nKJK, address:77.90.31.14:4696): protocol: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:56842->77.90.31.14:4696: use of closed network connection Nov 28 22:08:16.107260680 (420h21m40.874944104s) [noise=true , address=77.90.31.21:4907 , piece_id=RGBXOO3GD563J3Z7CY6DGGNLTO2WBZXUCMUWJYNQOCWJNJWMME5Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12RCGnjZC1UT9SUmFirNhS9pRBtxjcy91ss7c3Mk2kKnnHn2F6s, address:77.90.31.21:4907): protocol: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:54380->77.90.31.21:4907: use of closed network connection Nov 28 22:08:16.107466525 (420h21m40.87514995s) [noise=true , address=77.90.31.18:4825 , piece_id=VLCMFT3DSK7XUCD4XIZLNHLZP4PFNYVUD42OHVOXMKTZEG574FNQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:12PqKyfFxfVKTGARbPqBcQL3Naqjdf5uR44PeLi3sJpu57dcm1x, address:77.90.31.18:4825): protocol: read tcp 159.223.143.39:43194->77.90.31.18:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107641105 (420h21m40.87532453s) [noise=true , address=77.90.31.17:4784 , piece_id=YTVWWKOPFDVUV47E7QUNZCLBWJ2Y6ORHSXLN2PO3UN4RZLAVZHZA, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: read tcp 159.223.143.39:42966->77.90.31.17:4784: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107648798 (420h21m40.875332217s) [noise=true , address=45.91.116.7:4039 , piece_id=JNI5R6IZGZWBQDY53VB6X64CNDFPRIJERZ7SU3APRTX7PSACHLTQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:16oq1Yv7JfHrg4gdHfhwuMj4UsWeQQPayKV6Pj1NY2CgJhuPr2, address:45.91.116.7:4039): protocol: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:43532->45.91.116.7:4039: use of closed network connection Nov 28 22:08:16.107794752 (420h21m40.875478183s) [noise=true , address=45.91.116.14:4218 , piece_id=62ZVYH35F7G2BFGH76EPLTNOEK6GOMJBACHJ7LNX5PABZPOWNTQQ, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:1V3LqYCcYTkZLnhLVPagFLYTj2fm5HmBJiPLiQuQypwEQopn24, address:45.91.116.14:4218): protocol: read tcp 159.223.143.39:38252->45.91.116.14:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.107949069 (420h21m40.8756325s) [noise=true , address=45.91.116.10:4103 , piece_id=6IQM47MIEB6DVAF4EUPB6CP73LVXTLPCEIWKUFCCW7FAKJNZ4Z7Q, seg_pos={0 0}, upload=437481]: Done uploading piece. err: ecclient: upload failed (node:126w8AaZrrF8Y4uMP5yrGGadnMCCEiuP1s2CKg4XwkA7kPMvvS5, address:45.91.116.10:4103): protocol: read tcp 159.223.143.39:44752->45.91.116.10:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:16.108001397 (420h21m40.875684822s) [seg_pos={0 0}, upload=437481]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:16.133345284 (420h21m40.901028699s) [upload=437481]: Done uploading segments... Nov 28 22:08:16.133381504 (420h21m40.901064931s) [upload=437481]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 563.085246ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 518.75177ms trace: https://jaeger.dev.storj.io/trace/1c82198ceaf4021c FAIL (2025-11-28 22:22:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 28 22:07:30.943955390 (420h20m55.711638822s) [upload=437487]: Starting upload Nov 28 22:07:30.944043276 (420h20m55.711726692s) [upload=437487]: Done starting upload Nov 28 22:07:30.944309995 (420h20m55.711993422s) [upload=437487]: Uploading segments... Nov 28 22:07:30.944374479 (420h20m55.712057994s) [upload=437487]: Got next segment. Inline: false Nov 28 22:07:30.944437098 (420h20m55.712120524s) [upload=437487]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:30.944445775 (420h20m55.712129205s) [upload=437487]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:30.978454795 (420h20m55.746138221s) [seg_pos={0 0}, upload=437487]: Begin upload segment... Nov 28 22:07:30.978512973 (420h20m55.746196407s) [seg_pos={0 0}, upload=437487]: Done begin upload segment. Nov 28 22:07:30.978558745 (420h20m55.74624217s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.978572977 (420h20m55.746256411s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979312226 (420h20m55.746995652s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979378645 (420h20m55.747062057s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.979805759 (420h20m55.747489186s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980190464 (420h20m55.747873879s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980552955 (420h20m55.748236371s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.980992600 (420h20m55.748676015s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.981653640 (420h20m55.749337072s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.982275367 (420h20m55.749958783s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.983911490 (420h20m55.751594932s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.985497875 (420h20m55.75318129s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.986410527 (420h20m55.754093957s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.988477366 (420h20m55.756160804s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.989399745 (420h20m55.75708317s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:30.990324848 (420h20m55.758008279s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Uploading piece... Nov 28 22:07:31.061669157 (420h20m55.829352583s) [upload=437487]: Next returned nil segment Nov 28 22:07:31.061682224 (420h20m55.82936565s) [upload=437487]: Waiting for error group managing segments... Nov 28 22:07:31.076174182 (420h20m55.843857606s) [noise=true , address=145.79.227.14:4840 , piece_id=44DPQPF7PV25WZYGKUQMLWMDTY4X53Q4OWZRGBJOFKRSENDZZ2BQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.081773918 (420h20m55.849457343s) [noise=true , address=145.79.227.32:4836 , piece_id=OOXCGFR75YUYJEOKFXT55K5F4JHBGEA5Q6RYIFSCIQVQSLMJTR5A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.085985187 (420h20m55.853668613s) [noise=true , address=145.79.227.4:4186 , piece_id=JM7N6VXQI422ZHD6UYU7VXPYNJMVPLMMLWFUJO5UES3FWINJWZMA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.087240702 (420h20m55.854924129s) [noise=true , address=145.79.227.24:4583 , piece_id=QRBGPUL4IWNGT5IQUAQ4M536J7MEF3H7LOW6QMEMQCWQHZ7N3MBA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.088477984 (420h20m55.856161399s) [noise=true , address=145.79.227.16:4910 , piece_id=Y4AAWZ67ZK5B32US6RXCMWX47IKRKJXEXCZMUCRDXAWL2UUACIOQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.089919416 (420h20m55.85760285s) [noise=true , address=145.79.227.28:4726 , piece_id=YUU6MS5OU6KE6CXRDV4A6SCDYOXCYGK2NS46NEAEBBP3ND7KDZWQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.090307792 (420h20m55.857991218s) [noise=true , address=145.79.227.7:4294 , piece_id=ZUYTWW5ECRLNXCNOPQNHO5R2YBOKKY4H772EHW5VRK26Q4UYUC4A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094146123 (420h20m55.861829548s) [noise=true , address=145.79.227.43:4157 , piece_id=7YFFZ56S27V5DBT3UN7R37CO2JDXQHBD2KIGFYE7AWSIMPVEIFAA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.094979069 (420h20m55.862662498s) [noise=true , address=145.79.227.21:4474 , piece_id=5NZHHPICRD53GJR34UY7AEVG3VTCHF6NCK2APAH3NOLE7JRZ36WA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.096798027 (420h20m55.864481452s) [noise=true , address=145.79.227.31:4798 , piece_id=7CIPJLGOBQNT43KGNMEWUO4HSCOFSD555HD423V4UFQIVOF2H3PA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.106832236 (420h20m55.874515665s) [noise=true , address=145.79.227.40:4049 , piece_id=YNNGTKQ3B5QDBUCK7GSZMZIRRZA76RC5OSPCRLTURS4DLXC3VVCQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:07:31.387525033 (420h20m56.155208458s) [noise=true , address=145.79.227.10:4403 , piece_id=D6WX4J5IN5XF3YOUCB3EJVSFW22GJLIQL2RFKVGLP5U2X7S3YGVQ, seg_pos={0 0}, upload=437487]: Done uploading piece. err: <nil> Nov 28 22:08:00.912946147 (420h21m25.680629573s) [noise=true , address=45.91.116.6:4973 , piece_id=GIIGAQATMGUU2QPM4UIYKYRLDLK4X3BCEJV5T4GXDMBEC6TIVINA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 159.223.143.39:59790->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913149496 (420h21m25.680832924s) [noise=true , address=45.91.116.8:4045 , piece_id=BXNSANZ3DVMYS6H6EWNV7E2NL3TK6JU4BUJLN42NL3MMFKRL3J5Q, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 159.223.143.39:60286->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913180790 (420h21m25.680864218s) [noise=true , address=45.91.116.4:4939 , piece_id=GOM2YJVHTOD2SU2A7PCQCRB7X6SKHBIPPCWODTC4XO3THAFU5S6A, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: read tcp 159.223.143.39:35216->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913201612 (420h21m25.680885031s) [noise=true , address=77.90.31.17:4771 , piece_id=JBU6DYJ6JGVGBCDFAOXGUMHO2UOZCMRRDN2SACJAZDF7KTZU3AOA, seg_pos={0 0}, upload=437487]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: rpc: dial tcp 77.90.31.17:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:00.913227663 (420h21m25.680911088s) [seg_pos={0 0}, upload=437487]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 28 22:08:00.936374434 (420h21m25.70405786s) [upload=437487]: Done uploading segments... Nov 28 22:08:00.936410460 (420h21m25.704093889s) [upload=437487]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.778146ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 402.876377ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:22:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 402.904057ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 408.943467ms trace: https://jaeger.dev.storj.io/trace/4d50ae4cf70b3510 FAIL (2025-11-28 22:22:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== Nov 28 22:07:15.839298984 (420h20m40.606982416s) [upload=437480]: Starting upload Nov 28 22:07:15.839446124 (420h20m40.607129549s) [upload=437480]: Done starting upload Nov 28 22:07:15.839520010 (420h20m40.607203437s) [upload=437480]: Uploading segments... Nov 28 22:07:15.839890324 (420h20m40.607573749s) [upload=437480]: Got next segment. Inline: false Nov 28 22:07:15.839927380 (420h20m40.607610808s) [upload=437480]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 28 22:07:15.839934601 (420h20m40.607618049s) [upload=437480]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 28 22:07:15.867631058 (420h20m40.635314486s) [seg_pos={0 0}, upload=437480]: Begin upload segment... Nov 28 22:07:15.867728528 (420h20m40.635411953s) [seg_pos={0 0}, upload=437480]: Done begin upload segment. Nov 28 22:07:15.867777026 (420h20m40.635460456s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868566598 (420h20m40.636250023s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.868871051 (420h20m40.636554479s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869246789 (420h20m40.636930225s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869586926 (420h20m40.637270355s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869714715 (420h20m40.637398143s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.869975948 (420h20m40.637659443s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870382371 (420h20m40.638065921s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870392669 (420h20m40.638076086s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870759767 (420h20m40.638443194s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.870840433 (420h20m40.638523864s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871180620 (420h20m40.638864055s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871377813 (420h20m40.63906124s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.871835408 (420h20m40.639518835s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872357649 (420h20m40.640041065s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872854717 (420h20m40.640538144s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.872907619 (420h20m40.640591044s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.873294877 (420h20m40.640978294s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Uploading piece... Nov 28 22:07:15.940694221 (420h20m40.70837764s) [upload=437480]: Next returned nil segment Nov 28 22:07:15.940748468 (420h20m40.708431895s) [upload=437480]: Waiting for error group managing segments... Nov 28 22:07:15.946909206 (420h20m40.714592634s) [noise=true , address=145.79.227.19:4089 , piece_id=JCJ56K4ZLCQ4XJGINUTKYSXNAWRXTP2LNGPFWUZVX3VFTNENMLVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947594566 (420h20m40.715277998s) [noise=true , address=145.79.227.24:4611 , piece_id=5OCH7C6BTQ5NSWKBRNWDJO2HDQGSPWYYQMSUSBYOOICQTZ4KOGMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.947805246 (420h20m40.715488675s) [noise=true , address=145.79.227.41:4089 , piece_id=S2LOZMUGY2HUE2ZUO7U5RVX42KAFZBBTPLZMR374ANW4BO7UDCXQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948237151 (420h20m40.715920578s) [noise=true , address=145.79.227.49:4366 , piece_id=G5XLKEFWL3S243TB5BXF7OVHTWJOXBXXZ7NZKOYQU5Z7SMBLH7EA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948459079 (420h20m40.716142505s) [noise=true , address=145.79.227.44:4200 , piece_id=E2GUKH6TNWU67XIVYSVPWAAEY7J72REZREZEUPWBB2XVHRQBIZ7Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.948753768 (420h20m40.716437194s) [noise=true , address=145.79.227.13:4829 , piece_id=OLTFI2F2BUOEQPRKOBL5JCS3OO5UZB6V2KP22KWOJ5B4O7NETAKA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:15.949270445 (420h20m40.716953864s) [noise=true , address=145.79.227.15:4878 , piece_id=ZRB3ASRPBB3TUW5KHGA54VAAY5XGLMHKBNPTBYW3HEBPPISZKHPQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: <nil> Nov 28 22:07:16.263258655 (420h20m41.030942085s) [noise=true , address=64.49.13.169:4716 , piece_id=T5URAD6UCOFUL2GENMH4GIKEUCIEK2XRL52K6U6GP7I6RAYZICQA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN, address:64.49.13.169:4716): protocol: storage node overloaded, request limit: 2000 Nov 28 22:08:15.796481741 (420h21m40.564165171s) [noise=true , address=45.91.116.10:4109 , piece_id=USLMLAVY5LOK3IEZ74SSW6W5MQQW2HKHLQAR26HM4RTRASM4NRVA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12KxvKeR7QALkoH9b9ttiTwTunhNiW2ez95rsZ1xe2vmPG8BXCa, address:45.91.116.10:4109): protocol: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:33192->45.91.116.10:4109: use of closed network connection Nov 28 22:08:15.796644219 (420h21m40.564327648s) [noise=true , address=77.90.31.12:4623 , piece_id=HL2PYTB3KR3G4GDQVXFSYSI57MYVCHD72T2UMN3II6UQIH33K2GA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1ANjFmZrb9SDcqEHW8CBzeMtwvmceoLH9MtPabAjBw8wXjBuaM, address:77.90.31.12:4623): protocol: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:55292->77.90.31.12:4623: use of closed network connection Nov 28 22:08:15.796740455 (420h21m40.564423879s) [noise=true , address=77.90.31.19:4847 , piece_id=YQGRL3PO64CTXRFLVESWIKPQPUCDONNOT5B3YMPLAQV2SDQYZ5AA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12RkRJoWfBB5tUsZScQgydJhY5XffVuQn1JUa1ZHThjcP3bUfJv, address:77.90.31.19:4847): protocol: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:50080->77.90.31.19:4847: use of closed network connection Nov 28 22:08:15.796881828 (420h21m40.564565257s) [noise=true , address=77.90.31.18:4840 , piece_id=3MNOMXCCB56B4VUKMMKIE2RXMYVT6BKBYY2UIJL2SKUO23MBRSOQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:49610->77.90.31.18:4840: use of closed network connection Nov 28 22:08:15.797127972 (420h21m40.5648114s) [noise=true , address=45.91.116.9:4275 , piece_id=2YHMCJLDRJQSQRUR3NXX7FM3RFY2T5DC7GRIAXVHZR4SVVQNBQMA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:12S3ShBLc5k6bJJVNB6p68zF4kWAyVfDbF2stUyZngzTNtaEGz8, address:45.91.116.9:4275): protocol: read tcp 159.223.143.39:59426->45.91.116.9:4275: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797229901 (420h21m40.564913319s) [noise=true , address=77.90.31.21:4882 , piece_id=JWH4RPZQPLSDUXEFQUMGZSBEZFZ2RG5IUO2WW26FWI7DLN3RPU3Q, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1qmk23WxPsi88atKkdoQN3vJ3K1YDfNHYBMgYzcc52VURVA9V1, address:77.90.31.21:4882): protocol: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection; piecestore: piecestore close: write tcp 159.223.143.39:34134->77.90.31.21:4882: use of closed network connection Nov 28 22:08:15.797322173 (420h21m40.565005602s) [noise=true , address=45.91.116.13:4169 , piece_id=ET4VJIXHFWBHHEBWMMJ3TJVXK2KSQHDMQ2X5SEPMTXVPERUHKUVQ, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1c22WNWB9JrtWJ8yE2xo38kRbsrNiXRHy8opfwHuKMmymrwX9g, address:45.91.116.13:4169): protocol: read tcp 159.223.143.39:33474->45.91.116.13:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797511853 (420h21m40.565195284s) [noise=true , address=77.90.31.11:4574 , piece_id=N2BIZ5MIDHRBHJ2WP2OSXWGFY2GWXUMCG24OXIYHZX7RVGL3VAEA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, address:77.90.31.11:4574): protocol: read tcp 159.223.143.39:38440->77.90.31.11:4574: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797731173 (420h21m40.565414605s) [noise=true , address=77.90.31.15:4723 , piece_id=P6ARP526NIHYQNK7AGTUU4GTLS6ZCLB64I2BOV22BRLY5QVDZWXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS, address:77.90.31.15:4723): protocol: read tcp 159.223.143.39:59304->77.90.31.15:4723: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797792016 (420h21m40.565475497s) [noise=true , address=77.90.31.17:4789 , piece_id=CQGMLYH3AYERS27VWOPAA557TUINEYEGUOQ3W3PCVNDI56O3WGXA, seg_pos={0 0}, upload=437480]: Done uploading piece. err: ecclient: upload failed (node:1yBCEYPaZNi4HA8Kt6JPgnAtZ4FCkrEs28yTknjmBpG6A6wCTx, address:77.90.31.17:4789): protocol: read tcp 159.223.143.39:59214->77.90.31.17:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 28 22:08:15.797840872 (420h21m40.565524289s) [seg_pos={0 0}, upload=437480]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Nov 28 22:08:15.817177482 (420h21m40.584860911s) [upload=437480]: Done uploading segments... Nov 28 22:08:15.817218891 (420h21m40.584902317s) [upload=437480]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 408.97226ms core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)